[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:53 dist-test-slave-40vx krb5kdc[9017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:53 dist-test-slave-40vx krb5kdc[9017](info): set up 2 sockets
Sep 27 22:52:53 dist-test-slave-40vx krb5kdc[9017](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:52:54.052593 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:52:54 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:52:54.223340 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40387
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:54.531262 9030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:54.531634 9030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:54.531802 9030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:54.538875 9030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:54.539026 9030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:54.539135 9030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:54.539239 9030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:54.539343 9030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:54.548879 9030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:54.550230 9030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:54.551558 9030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:54.563638 9035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:54.563620 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:54.563669 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:54.563933 9030 server_base.cc:1034] running on GCE node
I20260927 22:52:54.565078 9030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:54.566365 9030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:54.567554 9030 hybrid_clock.cc:648] HybridClock initialized: now 1790549574567500 us; error 41 us; skew 500 ppm
I20260927 22:52:54.567904 9030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:54 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549574, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:54.575700 9030 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:52:54.577175 9030 webserver.cc:460] Webserver started at http://127.8.204.190:44865/ using document root <none> and password file <none>
I20260927 22:52:54.577805 9030 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:54.577899 9030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:54.578133 9030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:54.581549 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "67573819ee1d409c90872374fdebff25"
format_stamp: "Formatted at 2026-09-27 22:52:54 on dist-test-slave-40vx"
server_key: "246b5d50ef322a0bce91a4560a0c8869"
server_key_iv: "661ac494d46f8deef0d537654fa1f37e"
server_key_version: "encryptionkey@0"
I20260927 22:52:54.582593 9030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "67573819ee1d409c90872374fdebff25"
format_stamp: "Formatted at 2026-09-27 22:52:54 on dist-test-slave-40vx"
server_key: "246b5d50ef322a0bce91a4560a0c8869"
server_key_iv: "661ac494d46f8deef0d537654fa1f37e"
server_key_version: "encryptionkey@0"
I20260927 22:52:54.587826 9030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:52:54.591046 9045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:54.592379 9030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:54.592566 9030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "67573819ee1d409c90872374fdebff25"
format_stamp: "Formatted at 2026-09-27 22:52:54 on dist-test-slave-40vx"
server_key: "246b5d50ef322a0bce91a4560a0c8869"
server_key_iv: "661ac494d46f8deef0d537654fa1f37e"
server_key_version: "encryptionkey@0"
I20260927 22:52:54.592764 9030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:54.608111 9030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:54.612052 9030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:54.612336 9030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:54.630707 9030 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40387
I20260927 22:52:54.630735 9096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40387 every 8 connection(s)
I20260927 22:52:54.632158 9030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:54.635906 9097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:52:54.638630 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9030
I20260927 22:52:54.638932 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:52:54.640746 9010 external_mini_cluster.cc:1428] Setting key 0e41777ac5180021e4bb8e7c2026a243
I20260927 22:52:54.646797 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25: Bootstrap starting.
I20260927 22:52:54.649608 9097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:54.650585 9097 log.cc:824] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:54.653329 9097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25: No bootstrap required, opened a new log
I20260927 22:52:54.661062 9097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67573819ee1d409c90872374fdebff25" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40387 } }
I20260927 22:52:54.661439 9097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:54.661556 9097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67573819ee1d409c90872374fdebff25, State: Initialized, Role: FOLLOWER
I20260927 22:52:54.662184 9097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67573819ee1d409c90872374fdebff25" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40387 } }
I20260927 22:52:54.662436 9097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:54.662600 9097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:54.662797 9097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:54.666203 9097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67573819ee1d409c90872374fdebff25" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40387 } }
I20260927 22:52:54.666911 9097 leader_election.cc:302] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67573819ee1d409c90872374fdebff25; no voters:
I20260927 22:52:54.667747 9097 leader_election.cc:288] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:54.667852 9102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:54.668535 9102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [term 1 LEADER]: Becoming Leader. State: Replica: 67573819ee1d409c90872374fdebff25, State: Running, Role: LEADER
Sep 27 22:52:54 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:52:54.669139 9102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67573819ee1d409c90872374fdebff25" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40387 } }
I20260927 22:52:54.670733 9097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:54.672710 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67573819ee1d409c90872374fdebff25. Latest consensus state: current_term: 1 leader_uuid: "67573819ee1d409c90872374fdebff25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67573819ee1d409c90872374fdebff25" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40387 } } }
I20260927 22:52:54.673103 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:54.674805 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67573819ee1d409c90872374fdebff25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67573819ee1d409c90872374fdebff25" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40387 } } }
I20260927 22:52:54.675259 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:54.677968 9107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:54.682377 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:54.644482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.8.204.190:40387)
0927 22:52:54.645212 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:54.645246 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:54.646370 (+ 1124us) server_negotiation.cc:407] Connection header received
0927 22:52:54.654998 (+ 8628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:54.655018 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:54.655515 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:54.655943 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:54.658693 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:54.660111 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:54.661878 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:54.662662 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:54.662846 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:54.670441 (+ 7595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:54.670459 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:54.670486 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:54.670557 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:54.675770 (+ 5213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:54.677136 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:54.677146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:54.677158 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:54.677391 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:54.678076 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:54.678083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:54.678087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:54.678934 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:54.679234 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:54.680471 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0927 22:52:54.680882 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":346,"threads_started":1}
I20260927 22:52:54.686272 9107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:54.695856 9107 catalog_manager.cc:1347] Generated new cluster ID: 99bba13e7c3a44e2bb50fccfa0b24bdf
I20260927 22:52:54.696331 9107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:54.707715 9107 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:54.709568 9107 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:54.720666 9107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67573819ee1d409c90872374fdebff25: Generated new TSK 0
I20260927 22:52:54.721689 9107 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:52:54.766695 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40387
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:55.093925 9123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:55.094218 9123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:55.094381 9123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:55.101588 9123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:55.101727 9123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:55.101910 9123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:52:55.111188 9123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40387
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:55.112440 9123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:55.113736 9123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:55.126909 9129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:55.126957 9128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:55.126935 9131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:55.128057 9123 server_base.cc:1034] running on GCE node
I20260927 22:52:55.128669 9123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:55.129550 9123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:55.130788 9123 hybrid_clock.cc:648] HybridClock initialized: now 1790549575130735 us; error 45 us; skew 500 ppm
I20260927 22:52:55.131091 9123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:55 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:55.134203 9123 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:52:55.135658 9123 webserver.cc:460] Webserver started at http://127.8.204.129:34449/ using document root <none> and password file <none>
I20260927 22:52:55.136260 9123 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:55.136356 9123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:55.136615 9123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:55.139343 9123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "59379b25982149728e54e5e95bfe182a"
format_stamp: "Formatted at 2026-09-27 22:52:55 on dist-test-slave-40vx"
server_key: "18497cd158ebcd51e4b6d2e8b404dd16"
server_key_iv: "dd9ce27d1b5e4432003424d5c4c05b26"
server_key_version: "encryptionkey@0"
I20260927 22:52:55.140041 9123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "59379b25982149728e54e5e95bfe182a"
format_stamp: "Formatted at 2026-09-27 22:52:55 on dist-test-slave-40vx"
server_key: "18497cd158ebcd51e4b6d2e8b404dd16"
server_key_iv: "dd9ce27d1b5e4432003424d5c4c05b26"
server_key_version: "encryptionkey@0"
I20260927 22:52:55.144665 9123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:52:55.147790 9138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:55.149012 9123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:55.149135 9123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "59379b25982149728e54e5e95bfe182a"
format_stamp: "Formatted at 2026-09-27 22:52:55 on dist-test-slave-40vx"
server_key: "18497cd158ebcd51e4b6d2e8b404dd16"
server_key_iv: "dd9ce27d1b5e4432003424d5c4c05b26"
server_key_version: "encryptionkey@0"
I20260927 22:52:55.149286 9123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:55.165658 9123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:55.169842 9123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:55.170068 9123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:55.171016 9123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:55.172459 9123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:55.172533 9123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:55.172626 9123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:55.172668 9123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:55.205294 9123 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:43233
I20260927 22:52:55.205331 9250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:43233 every 8 connection(s)
I20260927 22:52:55.206537 9123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:55.213768 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9123
I20260927 22:52:55.213992 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:55.214803 9010 external_mini_cluster.cc:1428] Setting key 326356fb72c1e77bce9cf8c29e2ef73c
Sep 27 22:52:55 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:52:55.226691 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:55.208879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:40105 (local address 127.8.204.190:40387)
0927 22:52:55.209311 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:55.209315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:55.210114 (+ 799us) server_negotiation.cc:407] Connection header received
0927 22:52:55.211925 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:55.211931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:55.212020 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:55.212137 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:55.213625 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.214471 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.216312 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.216740 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.216888 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:55.220582 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:55.220592 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:55.220597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:55.220641 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:55.223508 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:55.224400 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:55.224404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:55.224406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:55.224511 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:55.224967 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:55.224970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:55.224972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:55.225213 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:55.225315 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:55.226186 (+ 871us) server_negotiation.cc:299] Negotiation successful
0927 22:52:55.226353 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":231,"threads_started":1}
I20260927 22:52:55.228264 9254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:55.209290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40387 (local address 127.8.204.129:40105)
0927 22:52:55.209951 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:55.209990 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:55.211670 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:55.212364 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:55.212372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:55.212843 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:55.213434 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:55.213463 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.214633 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.214641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:55.216167 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:55.216179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.216961 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.216969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:55.217210 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:55.217869 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:55.217890 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:55.220324 (+ 2434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:55.223648 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:55.223659 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:55.223677 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:55.224278 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:55.224619 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:55.224626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:55.224633 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:55.224860 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:55.225336 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:55.225347 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:55.225797 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 22:52:55.226098 (+ 301us) client_negotiation.cc:240] Negotiation successful
0927 22:52:55.226707 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":267,"threads_started":1}
I20260927 22:52:55.230154 9251 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40387
I20260927 22:52:55.230458 9251 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:52:55.231209 9251 heartbeater.cc:511] Master 127.8.204.190:40387 requested a full tablet report, sending...
I20260927 22:52:55.233196 9062 ts_manager.cc:194] Registered new tserver with Master: 59379b25982149728e54e5e95bfe182a (127.8.204.129:43233)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:52:55.234774 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:40105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:52:55.254818 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40387
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:55.567658 9257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:55.567931 9257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:55.568056 9257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:55.575325 9257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:55.575443 9257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:55.575557 9257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:52:55.586419 9257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40387
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:55.587968 9257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:55.589342 9257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:55.602864 9263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:55.602851 9265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:55.602865 9262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:55.604219 9257 server_base.cc:1034] running on GCE node
I20260927 22:52:55.604763 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:55.605742 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:55.606958 9257 hybrid_clock.cc:648] HybridClock initialized: now 1790549575606907 us; error 60 us; skew 500 ppm
I20260927 22:52:55.607314 9257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:55 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:55.610272 9257 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:52:55.611807 9257 webserver.cc:460] Webserver started at http://127.8.204.130:36075/ using document root <none> and password file <none>
I20260927 22:52:55.612466 9257 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:55.612609 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:55.612911 9257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:55.615780 9257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "b21f0955443d479b9c5e594b2f58333f"
format_stamp: "Formatted at 2026-09-27 22:52:55 on dist-test-slave-40vx"
server_key: "1afe41646083f60ddba8b7c6798cfd03"
server_key_iv: "435b1428ee621e09b2abd85dea4d00a2"
server_key_version: "encryptionkey@0"
I20260927 22:52:55.616545 9257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "b21f0955443d479b9c5e594b2f58333f"
format_stamp: "Formatted at 2026-09-27 22:52:55 on dist-test-slave-40vx"
server_key: "1afe41646083f60ddba8b7c6798cfd03"
server_key_iv: "435b1428ee621e09b2abd85dea4d00a2"
server_key_version: "encryptionkey@0"
I20260927 22:52:55.621527 9257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:52:55.624934 9272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:55.626313 9257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:52:55.626505 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "b21f0955443d479b9c5e594b2f58333f"
format_stamp: "Formatted at 2026-09-27 22:52:55 on dist-test-slave-40vx"
server_key: "1afe41646083f60ddba8b7c6798cfd03"
server_key_iv: "435b1428ee621e09b2abd85dea4d00a2"
server_key_version: "encryptionkey@0"
I20260927 22:52:55.626721 9257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:55.654964 9257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:55.659027 9257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:55.659375 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:55.660329 9257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:55.661617 9257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:55.661722 9257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:55.661862 9257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:55.661934 9257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:55.693921 9257 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:44007
I20260927 22:52:55.693962 9384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:44007 every 8 connection(s)
I20260927 22:52:55.695210 9257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:55.700634 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9257
I20260927 22:52:55.700829 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:55.701346 9010 external_mini_cluster.cc:1428] Setting key 30d46b4e4aa9dc27f1829dec53a6d729
Sep 27 22:52:55 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549575, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:52:55.717343 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:55.697845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:48613 (local address 127.8.204.190:40387)
0927 22:52:55.698073 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:55.698079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:55.699193 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 22:52:55.701477 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:55.701482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:55.701552 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:55.701640 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:55.703512 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.704249 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.707148 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.707772 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.707914 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:55.711651 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:55.711656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:55.711659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:55.711688 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:55.714317 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:55.715044 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:55.715051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:55.715059 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:55.715178 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:55.715627 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:55.715632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:55.715635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:55.715894 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:55.716063 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:55.716773 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 22:52:55.717001 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:52:55.718528 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:55.698349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40387 (local address 127.8.204.130:48613)
0927 22:52:55.699035 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:55.699076 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:55.701149 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:55.701918 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:55.701931 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:55.702594 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:55.703311 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:55.703341 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.704423 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.704432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:55.706699 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:55.706716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:55.707979 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:55.707984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:55.708262 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:55.708942 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:55.708969 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:55.711425 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:55.714464 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:55.714472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:55.714489 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:55.714910 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:55.715314 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:55.715319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:55.715323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:55.715504 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:55.716040 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:55.716047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:55.716406 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 22:52:55.716678 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 22:52:55.717228 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":268,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
I20260927 22:52:55.720062 9385 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40387
I20260927 22:52:55.720363 9385 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:55.721030 9385 heartbeater.cc:511] Master 127.8.204.190:40387 requested a full tablet report, sending...
I20260927 22:52:55.722553 9062 ts_manager.cc:194] Registered new tserver with Master: b21f0955443d479b9c5e594b2f58333f (127.8.204.130:44007)
I20260927 22:52:55.723608 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:48613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:52:55.740835 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40387
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:56.057204 9390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:56.057471 9390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:56.057624 9390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:56.064734 9390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:56.064831 9390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:56.064997 9390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:52:56.074573 9390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40387
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:56.075901 9390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:56.077198 9390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:56.090168 9395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:56.090246 9396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:56.090209 9398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:56.091135 9390 server_base.cc:1034] running on GCE node
I20260927 22:52:56.091784 9390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:56.092717 9390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:56.093921 9390 hybrid_clock.cc:648] HybridClock initialized: now 1790549576093874 us; error 36 us; skew 500 ppm
I20260927 22:52:56.094259 9390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:56.097649 9390 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:52:56.099143 9390 webserver.cc:460] Webserver started at http://127.8.204.131:36803/ using document root <none> and password file <none>
I20260927 22:52:56.099869 9390 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:56.099965 9390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:56.100221 9390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:56.102906 9390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "1dd22c345a354a5995d9d7173eb06075"
format_stamp: "Formatted at 2026-09-27 22:52:56 on dist-test-slave-40vx"
server_key: "ed1e52c5ff2de9f7bec962cedd2dfdd7"
server_key_iv: "a3f486bfd7da84518a17a8f2934111b4"
server_key_version: "encryptionkey@0"
I20260927 22:52:56.103662 9390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "1dd22c345a354a5995d9d7173eb06075"
format_stamp: "Formatted at 2026-09-27 22:52:56 on dist-test-slave-40vx"
server_key: "ed1e52c5ff2de9f7bec962cedd2dfdd7"
server_key_iv: "a3f486bfd7da84518a17a8f2934111b4"
server_key_version: "encryptionkey@0"
I20260927 22:52:56.108536 9390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:52:56.111872 9405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:56.113193 9390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:52:56.113356 9390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "1dd22c345a354a5995d9d7173eb06075"
format_stamp: "Formatted at 2026-09-27 22:52:56 on dist-test-slave-40vx"
server_key: "ed1e52c5ff2de9f7bec962cedd2dfdd7"
server_key_iv: "a3f486bfd7da84518a17a8f2934111b4"
server_key_version: "encryptionkey@0"
I20260927 22:52:56.113547 9390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:56.130228 9390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:56.134577 9390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:56.134861 9390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:56.135840 9390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:56.137152 9390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:56.137254 9390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:56.137400 9390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:56.137477 9390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:56.169801 9390 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36307
I20260927 22:52:56.169842 9517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36307 every 8 connection(s)
I20260927 22:52:56.171202 9390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:56.176071 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9390
I20260927 22:52:56.176266 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:56.176782 9010 external_mini_cluster.cc:1428] Setting key c73478efd507c3dd94e348e4f707d7fd
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:52:56.190986 9253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.173688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:40463 (local address 127.8.204.190:40387)
0927 22:52:56.173888 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:56.173894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:56.175122 (+ 1228us) server_negotiation.cc:407] Connection header received
0927 22:52:56.177677 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:56.177683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:56.177801 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:56.177905 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:56.179850 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.180659 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.182153 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.182560 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.182690 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.185924 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:56.185928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:56.185931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:56.185960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:56.188493 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.189156 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.189160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.189163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.189269 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.189640 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.189643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.189645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.189886 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:56.189988 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:56.190557 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 22:52:56.190714 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:52:56.192214 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.174234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40387 (local address 127.8.204.131:40463)
0927 22:52:56.174949 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:56.174994 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:56.177347 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:56.178179 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:56.178188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:56.178844 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:56.179697 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:56.179716 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.180804 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.180809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:56.182008 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:56.182019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.182770 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.182778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:56.183025 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.183651 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:56.183672 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:56.185740 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:56.188642 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:56.188649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:56.188659 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:56.189051 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:56.189355 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:56.189358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:56.189360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:56.189525 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:56.189982 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:56.189986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:56.190269 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 22:52:56.190475 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:52:56.190964 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":289,"threads_started":1}
I20260927 22:52:56.193583 9518 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40387
I20260927 22:52:56.193876 9518 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:56.194573 9518 heartbeater.cc:511] Master 127.8.204.190:40387 requested a full tablet report, sending...
I20260927 22:52:56.195955 9062 ts_manager.cc:194] Registered new tserver with Master: 1dd22c345a354a5995d9d7173eb06075 (127.8.204.131:36307)
I20260927 22:52:56.196830 9062 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:40463
I20260927 22:52:56.205122 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260927 22:52:56.238345 9251 heartbeater.cc:503] Master 127.8.204.190:40387 was elected leader, sending a full tablet report...
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:52:56.248684 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.233478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.8.204.129:43233)
0927 22:52:56.234232 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:56.234239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:56.234267 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:52:56.234331 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:56.234338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:56.234569 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:56.234706 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:56.236193 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.237219 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.238626 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.239265 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.239425 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.243385 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:56.243396 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:56.243411 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:56.243441 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:56.246254 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.246901 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.246908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.246910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.247040 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.247386 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.247389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.247391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.247716 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:56.247926 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:56.248198 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:52:56.248389 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":447,"threads_started":1}
W20260927 22:52:56.249882 9166 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56872
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:52:56.292287 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.278391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.8.204.129:43233)
0927 22:52:56.278692 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:56.278699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:56.279015 (+ 316us) server_negotiation.cc:407] Connection header received
0927 22:52:56.279175 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:56.279180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:56.279247 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:56.279433 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:56.280770 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.281503 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.282769 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.283186 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.283361 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.287163 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:56.287169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:56.287172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:56.287205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:56.289834 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.290474 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.290478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.290481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.290604 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.291005 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.291009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.291011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.291263 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:56.291527 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:56.291868 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 22:52:56.292041 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:52:56.293807 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9123
I20260927 22:52:56.306085 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.294848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.8.204.129:43233)
0927 22:52:56.295043 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:56.295051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:56.295174 (+ 123us) server_negotiation.cc:407] Connection header received
0927 22:52:56.295338 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:56.295341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:56.295396 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:56.295470 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:56.296615 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.297321 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.298634 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.299038 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.299126 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.301442 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:56.301447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:56.301450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:56.301477 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:56.304017 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.304575 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.304578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.304581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.304704 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.304981 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.304984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.304986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.305202 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:56.305376 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:56.305584 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:52:56.305725 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:52:56.306783 9246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:56.389719 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9123
I20260927 22:52:56.405650 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9257
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:52:56.420919 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.406504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.8.204.130:44007)
0927 22:52:56.407074 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:56.407080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:56.407106 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:52:56.407166 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:56.407172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:56.407435 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:56.407584 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:56.408610 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.409599 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.410735 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.411261 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.411494 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.415232 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:56.415241 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:56.415254 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:56.415334 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:56.418312 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.418889 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.418895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.418897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.419046 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.419496 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.419501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.419504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.419827 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:56.420037 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:56.420346 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:52:56.420525 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":290,"threads_started":1}
I20260927 22:52:56.422106 9380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:56.504974 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9257
I20260927 22:52:56.521395 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9390
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:52:56.536639 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:56.522421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.8.204.131:36307)
0927 22:52:56.523011 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:56.523027 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:56.523056 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:52:56.523116 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:56.523122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:56.523428 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:56.523565 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:56.524732 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.525742 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.526944 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:56.527586 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:56.527686 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:56.531497 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:56.531507 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:56.531525 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:56.531560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:56.534266 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.534850 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.534856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.534858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.534966 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:56.535262 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:56.535264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:56.535266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:56.535619 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:56.535835 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:56.536057 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:52:56.536235 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":313,"threads_started":1}
I20260927 22:52:56.537864 9513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:56.621981 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9390
I20260927 22:52:56.637444 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9030
I20260927 22:52:56.638613 9092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:52:56.739537 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9030
2026-09-27T22:52:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3782 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9539](info): set up 2 sockets
Sep 27 22:52:56 dist-test-slave-40vx krb5kdc[9539](info): commencing operation
krb5kdc: starting...
W20260927 22:52:57.823185 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:52:57 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:52:57.991202 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37341
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:58.300356 9552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:58.300696 9552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:58.300850 9552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:58.307951 9552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:52:58.308090 9552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:58.308197 9552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:58.308298 9552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:52:58.308382 9552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:52:58.317746 9552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:58.318997 9552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:58.320271 9552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:58.331545 9558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:58.331475 9560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:58.332055 9557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:58.332232 9552 server_base.cc:1034] running on GCE node
I20260927 22:52:58.333027 9552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:58.334267 9552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:58.335484 9552 hybrid_clock.cc:648] HybridClock initialized: now 1790549578335444 us; error 51 us; skew 500 ppm
I20260927 22:52:58.335839 9552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:58 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549578, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:58.338680 9552 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:52:58.340021 9552 webserver.cc:460] Webserver started at http://127.8.204.190:34919/ using document root <none> and password file <none>
I20260927 22:52:58.340605 9552 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:58.340696 9552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:58.340926 9552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:58.343616 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "9c029743a30242a5bb0e1f9e30184e72"
format_stamp: "Formatted at 2026-09-27 22:52:58 on dist-test-slave-40vx"
server_key: "2d01637c4589015734669135189fc1c0"
server_key_iv: "121a0438f8dd5b68e293df3f54ff5645"
server_key_version: "encryptionkey@0"
I20260927 22:52:58.344291 9552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "9c029743a30242a5bb0e1f9e30184e72"
format_stamp: "Formatted at 2026-09-27 22:52:58 on dist-test-slave-40vx"
server_key: "2d01637c4589015734669135189fc1c0"
server_key_iv: "121a0438f8dd5b68e293df3f54ff5645"
server_key_version: "encryptionkey@0"
I20260927 22:52:58.349032 9552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:52:58.352288 9567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:58.353564 9552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:52:58.353740 9552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "9c029743a30242a5bb0e1f9e30184e72"
format_stamp: "Formatted at 2026-09-27 22:52:58 on dist-test-slave-40vx"
server_key: "2d01637c4589015734669135189fc1c0"
server_key_iv: "121a0438f8dd5b68e293df3f54ff5645"
server_key_version: "encryptionkey@0"
I20260927 22:52:58.353940 9552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:58.370915 9552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:58.374724 9552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:58.375012 9552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:58.392903 9552 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:37341
I20260927 22:52:58.392911 9618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:37341 every 8 connection(s)
I20260927 22:52:58.394644 9552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:52:58.396380 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9552
I20260927 22:52:58.396612 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:52:58.397236 9010 external_mini_cluster.cc:1428] Setting key 072b49566fa32b7d1e4cbb1f32b5ebea
I20260927 22:52:58.398593 9619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:52:58 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:52:58.412158 9619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72: Bootstrap starting.
I20260927 22:52:58.416251 9619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72: Neither blocks nor log segments found. Creating new log.
I20260927 22:52:58.417450 9619 log.cc:824] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72: Log is configured to *not* fsync() on all Append() calls
I20260927 22:52:58.420490 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:58.399103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.8.204.190:37341)
0927 22:52:58.399936 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:58.399947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:58.399987 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:52:58.401490 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:58.401510 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:58.401896 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:58.402214 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:58.403897 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:58.405305 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:58.407013 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:58.407712 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:58.407833 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:58.412319 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:58.412329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:58.412346 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:58.412389 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:58.416105 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:58.416764 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:58.416771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:58.416779 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:58.416960 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:58.417293 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:58.417298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:58.417301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:58.417967 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:58.418246 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:58.418802 (+ 556us) server_negotiation.cc:299] Negotiation successful
0927 22:52:58.419202 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":435,"threads_started":1}
I20260927 22:52:58.420684 9619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72: No bootstrap required, opened a new log
I20260927 22:52:58.428328 9619 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c029743a30242a5bb0e1f9e30184e72" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37341 } }
I20260927 22:52:58.428623 9619 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:52:58.428694 9619 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c029743a30242a5bb0e1f9e30184e72, State: Initialized, Role: FOLLOWER
I20260927 22:52:58.429206 9619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c029743a30242a5bb0e1f9e30184e72" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37341 } }
I20260927 22:52:58.429387 9619 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:52:58.429474 9619 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:52:58.429625 9619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:52:58.431893 9619 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c029743a30242a5bb0e1f9e30184e72" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37341 } }
I20260927 22:52:58.432348 9619 leader_election.cc:302] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c029743a30242a5bb0e1f9e30184e72; no voters:
I20260927 22:52:58.432899 9619 leader_election.cc:288] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:52:58.433028 9624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:52:58.433681 9624 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [term 1 LEADER]: Becoming Leader. State: Replica: 9c029743a30242a5bb0e1f9e30184e72, State: Running, Role: LEADER
I20260927 22:52:58.434150 9624 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c029743a30242a5bb0e1f9e30184e72" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37341 } }
I20260927 22:52:58.434777 9619 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:52:58.436865 9626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c029743a30242a5bb0e1f9e30184e72. Latest consensus state: current_term: 1 leader_uuid: "9c029743a30242a5bb0e1f9e30184e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c029743a30242a5bb0e1f9e30184e72" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37341 } } }
I20260927 22:52:58.436934 9625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c029743a30242a5bb0e1f9e30184e72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c029743a30242a5bb0e1f9e30184e72" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37341 } } }
I20260927 22:52:58.437206 9626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:58.437321 9625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72 [sys.catalog]: This master's current role is: LEADER
I20260927 22:52:58.438958 9629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:52:58.443241 9629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:52:58.451138 9629 catalog_manager.cc:1347] Generated new cluster ID: 013cae23a87e418baced421beec02145
I20260927 22:52:58.451247 9629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:52:58.465530 9629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:52:58.467301 9629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:52:58.478451 9629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c029743a30242a5bb0e1f9e30184e72: Generated new TSK 0
I20260927 22:52:58.479432 9629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:52:58.522943 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37341
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:58.841674 9645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:58.841950 9645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:58.842049 9645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:58.849129 9645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:58.849227 9645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:58.849337 9645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:52:58.858868 9645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:58.860080 9645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:58.861285 9645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:58.874200 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:58.874109 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:58.874154 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:58.874723 9645 server_base.cc:1034] running on GCE node
I20260927 22:52:58.875377 9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:58.876209 9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:58.877439 9645 hybrid_clock.cc:648] HybridClock initialized: now 1790549578877398 us; error 53 us; skew 500 ppm
I20260927 22:52:58.877871 9645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:58 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549578, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:58.881156 9645 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:52:58.882793 9645 webserver.cc:460] Webserver started at http://127.8.204.129:45831/ using document root <none> and password file <none>
I20260927 22:52:58.883563 9645 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:58.883685 9645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:58.883997 9645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:58.886770 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "fe4bac5732ce421fbc0ddec82f1652af"
format_stamp: "Formatted at 2026-09-27 22:52:58 on dist-test-slave-40vx"
server_key: "017374cb1333ad2789ade6fa11db4de5"
server_key_iv: "208402bf813a30d0d1e8bf1d3d3ea259"
server_key_version: "encryptionkey@0"
I20260927 22:52:58.887619 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "fe4bac5732ce421fbc0ddec82f1652af"
format_stamp: "Formatted at 2026-09-27 22:52:58 on dist-test-slave-40vx"
server_key: "017374cb1333ad2789ade6fa11db4de5"
server_key_iv: "208402bf813a30d0d1e8bf1d3d3ea259"
server_key_version: "encryptionkey@0"
I20260927 22:52:58.892553 9645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:52:58.895982 9660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:58.897260 9645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:52:58.897418 9645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "fe4bac5732ce421fbc0ddec82f1652af"
format_stamp: "Formatted at 2026-09-27 22:52:58 on dist-test-slave-40vx"
server_key: "017374cb1333ad2789ade6fa11db4de5"
server_key_iv: "208402bf813a30d0d1e8bf1d3d3ea259"
server_key_version: "encryptionkey@0"
I20260927 22:52:58.897635 9645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:58.924288 9645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:58.928975 9645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:58.929258 9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:58.930213 9645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:58.931555 9645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:58.931632 9645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:58.931789 9645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:58.931847 9645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:58.964741 9645 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:33067
I20260927 22:52:58.964927 9772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:33067 every 8 connection(s)
I20260927 22:52:58.966069 9645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:52:58.968994 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9645
I20260927 22:52:58.969202 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:52:58.969796 9010 external_mini_cluster.cc:1428] Setting key 2b595ee13919870da387ccd03bf167cf
Sep 27 22:52:58 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549578, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:52:58.987205 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:58.968613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:49605 (local address 127.8.204.190:37341)
0927 22:52:58.969103 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:58.969110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:58.970237 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 22:52:58.972293 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:58.972301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:58.972381 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:58.972498 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:58.973880 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:58.975018 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:58.976561 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:58.977128 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:58.977254 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:58.981486 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:58.981492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:58.981496 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:58.981533 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:58.984067 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:58.984790 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:58.984800 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:58.984802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:58.984914 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:58.985329 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:58.985334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:58.985337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:58.985613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:58.985762 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:58.986590 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 22:52:58.986837 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":220,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:52:58.988364 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:58.969257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37341 (local address 127.8.204.129:49605)
0927 22:52:58.970055 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:58.970101 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:58.972032 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:58.972683 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:58.972690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:58.973176 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:58.973715 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:58.973731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:58.975156 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:58.975160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:58.976414 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:58.976425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:58.977371 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:58.977380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:58.977705 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:58.978475 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:58.978509 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:58.981276 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:58.984208 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:58.984217 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:58.984240 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:58.984663 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:58.985028 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:58.985032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:58.985036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:58.985212 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:58.985743 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:58.985751 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:58.986192 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 22:52:58.986531 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 22:52:58.987048 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":327,"threads_started":1}
I20260927 22:52:58.989871 9773 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37341
I20260927 22:52:58.990203 9773 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:58.991091 9773 heartbeater.cc:511] Master 127.8.204.190:37341 requested a full tablet report, sending...
I20260927 22:52:58.993503 9584 ts_manager.cc:194] Registered new tserver with Master: fe4bac5732ce421fbc0ddec82f1652af (127.8.204.129:33067)
I20260927 22:52:58.995703 9584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:49605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:52:59.011382 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37341
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:59.319628 9779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:59.319913 9779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:59.320081 9779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:59.327538 9779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:59.327648 9779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:59.327883 9779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:52:59.337958 9779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:59.339178 9779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:59.340472 9779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:59.353001 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:59.353077 9784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:59.353058 9785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:59.353820 9779 server_base.cc:1034] running on GCE node
I20260927 22:52:59.354332 9779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:59.355187 9779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:59.356374 9779 hybrid_clock.cc:648] HybridClock initialized: now 1790549579356342 us; error 23 us; skew 500 ppm
I20260927 22:52:59.356680 9779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:59 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:59.359782 9779 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:52:59.361200 9779 webserver.cc:460] Webserver started at http://127.8.204.130:37773/ using document root <none> and password file <none>
I20260927 22:52:59.361864 9779 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:59.361990 9779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:59.362298 9779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:59.365152 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "b127680b5d3f4f1c91b99fd9e98107f1"
format_stamp: "Formatted at 2026-09-27 22:52:59 on dist-test-slave-40vx"
server_key: "6b3f5b98329a58f8949fe41e2d0c15c6"
server_key_iv: "ff33f5880afa84400af35ca121dab304"
server_key_version: "encryptionkey@0"
I20260927 22:52:59.365921 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "b127680b5d3f4f1c91b99fd9e98107f1"
format_stamp: "Formatted at 2026-09-27 22:52:59 on dist-test-slave-40vx"
server_key: "6b3f5b98329a58f8949fe41e2d0c15c6"
server_key_iv: "ff33f5880afa84400af35ca121dab304"
server_key_version: "encryptionkey@0"
I20260927 22:52:59.370815 9779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:52:59.374112 9794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:59.375432 9779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:52:59.375597 9779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "b127680b5d3f4f1c91b99fd9e98107f1"
format_stamp: "Formatted at 2026-09-27 22:52:59 on dist-test-slave-40vx"
server_key: "6b3f5b98329a58f8949fe41e2d0c15c6"
server_key_iv: "ff33f5880afa84400af35ca121dab304"
server_key_version: "encryptionkey@0"
I20260927 22:52:59.375795 9779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:59.394225 9779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:59.398514 9779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:59.398808 9779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:59.399824 9779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:59.401098 9779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:59.401175 9779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:59.401342 9779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:59.401391 9779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:59.433122 9779 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:39845
I20260927 22:52:59.433158 9906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:39845 every 8 connection(s)
I20260927 22:52:59.434345 9779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:52:59.436277 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9779
I20260927 22:52:59.436458 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:52:59.436982 9010 external_mini_cluster.cc:1428] Setting key 411571b218b072d2beb5ce3407263fec
Sep 27 22:52:59 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:52:59.455487 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:59.436988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:49073 (local address 127.8.204.190:37341)
0927 22:52:59.437189 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:59.437195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:59.438590 (+ 1395us) server_negotiation.cc:407] Connection header received
0927 22:52:59.440561 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:59.440569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:59.440659 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:59.440763 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:59.442126 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.443387 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.445473 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.445865 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.445961 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:59.449907 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:59.449912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:59.449915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:59.449943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:59.452513 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:59.453222 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:59.453229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:59.453235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:59.453375 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:59.453845 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:59.453852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:59.453855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:59.454109 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:59.454253 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:59.454916 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 22:52:59.455125 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
I20260927 22:52:59.456668 9909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:59.437551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37341 (local address 127.8.204.130:49073)
0927 22:52:59.438406 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:59.438466 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:59.440256 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:59.440953 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:59.440961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:59.441431 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:59.441965 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:59.441981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.443555 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.443559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:59.445337 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:59.445354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.446085 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.446095 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:59.446412 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:59.447122 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:59.447161 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:59.449696 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:59.452681 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:59.452689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:59.452705 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:59.453095 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:59.453480 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:59.453485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:59.453489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:59.453741 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:59.454241 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:59.454248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:59.454578 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 22:52:59.454841 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:52:59.455389 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":356,"threads_started":1}
I20260927 22:52:59.458189 9907 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37341
I20260927 22:52:59.458540 9907 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:59.459463 9907 heartbeater.cc:511] Master 127.8.204.190:37341 requested a full tablet report, sending...
I20260927 22:52:59.461166 9584 ts_manager.cc:194] Registered new tserver with Master: b127680b5d3f4f1c91b99fd9e98107f1 (127.8.204.130:39845)
I20260927 22:52:59.462210 9584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:49073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:52:59.478874 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37341
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:52:59.791316 9912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:52:59.791581 9912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:52:59.791677 9912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:52:59.798738 9912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:52:59.798847 9912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:52:59.798969 9912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:52:59.811088 9912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:52:59.812700 9912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:52:59.814412 9912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:52:59.827242 9920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:59.827366 9918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:52:59.827405 9917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:52:59.828280 9912 server_base.cc:1034] running on GCE node
I20260927 22:52:59.828889 9912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:52:59.829814 9912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:52:59.831024 9912 hybrid_clock.cc:648] HybridClock initialized: now 1790549579830982 us; error 37 us; skew 500 ppm
I20260927 22:52:59.831395 9912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:52:59 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:52:59.838369 9912 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:52:59.839977 9912 webserver.cc:460] Webserver started at http://127.8.204.131:34769/ using document root <none> and password file <none>
I20260927 22:52:59.840708 9912 fs_manager.cc:359] Metadata directory not provided
I20260927 22:52:59.840834 9912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:52:59.841144 9912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:52:59.844069 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "374ead130a26425aa499a0cb2e3f7a35"
format_stamp: "Formatted at 2026-09-27 22:52:59 on dist-test-slave-40vx"
server_key: "99c0b9707e9799ad49ed1ab992bf060f"
server_key_iv: "72f289d883e85f4328307e1c1aace6fb"
server_key_version: "encryptionkey@0"
I20260927 22:52:59.844841 9912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "374ead130a26425aa499a0cb2e3f7a35"
format_stamp: "Formatted at 2026-09-27 22:52:59 on dist-test-slave-40vx"
server_key: "99c0b9707e9799ad49ed1ab992bf060f"
server_key_iv: "72f289d883e85f4328307e1c1aace6fb"
server_key_version: "encryptionkey@0"
I20260927 22:52:59.849961 9912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:52:59.853395 9927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:59.854687 9912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:52:59.854861 9912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "374ead130a26425aa499a0cb2e3f7a35"
format_stamp: "Formatted at 2026-09-27 22:52:59 on dist-test-slave-40vx"
server_key: "99c0b9707e9799ad49ed1ab992bf060f"
server_key_iv: "72f289d883e85f4328307e1c1aace6fb"
server_key_version: "encryptionkey@0"
I20260927 22:52:59.855064 9912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:52:59.884910 9912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:52:59.889298 9912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:52:59.889592 9912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:52:59.890549 9912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:52:59.891950 9912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:52:59.892050 9912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:59.892189 9912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:52:59.892266 9912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:52:59.924654 9912 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:34727
I20260927 22:52:59.924686 10039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:34727 every 8 connection(s)
I20260927 22:52:59.925974 9912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:52:59.934798 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 9912
I20260927 22:52:59.934990 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:52:59.935547 9010 external_mini_cluster.cc:1428] Setting key b3ea935a54bdb38763c73093b8952c25
Sep 27 22:52:59 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549579, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:52:59.945797 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:59.928419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:48859 (local address 127.8.204.190:37341)
0927 22:52:59.928628 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:59.928634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:59.929962 (+ 1328us) server_negotiation.cc:407] Connection header received
0927 22:52:59.931796 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:59.931800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:59.931866 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:59.931966 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:59.933351 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.934092 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.936360 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.936962 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.937070 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:59.940860 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:59.940864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:59.940868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:59.940898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:59.943296 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:59.943948 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:59.943951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:59.943961 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:59.944067 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:59.944434 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:59.944437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:59.944439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:59.944687 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:59.944790 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:59.945403 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:52:59.945561 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:52:59.946792 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:59.928971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37341 (local address 127.8.204.131:48859)
0927 22:52:59.929757 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 22:52:59.929803 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:52:59.931548 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:52:59.932176 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:52:59.932184 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:52:59.932655 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:52:59.933189 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:59.933205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.934230 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.934234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:59.936220 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:52:59.936231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.937203 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.937211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:52:59.937503 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:59.938271 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:52:59.938293 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:52:59.940666 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:52:59.943442 (+ 2776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:59.943448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:59.943460 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:59.943844 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:59.944157 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:52:59.944160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:52:59.944162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:52:59.944336 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:52:59.944785 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:52:59.944791 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:52:59.945089 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 22:52:59.945318 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 22:52:59.945700 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":350,"threads_started":1}
I20260927 22:52:59.948146 10040 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37341
I20260927 22:52:59.948426 10040 heartbeater.cc:464] Registering TS with master...
I20260927 22:52:59.949175 10040 heartbeater.cc:511] Master 127.8.204.190:37341 requested a full tablet report, sending...
I20260927 22:52:59.950703 9584 ts_manager.cc:194] Registered new tserver with Master: 374ead130a26425aa499a0cb2e3f7a35 (127.8.204.131:34727)
I20260927 22:52:59.951520 9584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:48859
I20260927 22:52:59.962934 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:52:59.981076 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:52:59.969252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.8.204.190:37341)
0927 22:52:59.969466 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:52:59.969475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:52:59.969960 (+ 485us) server_negotiation.cc:407] Connection header received
0927 22:52:59.970082 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:52:59.970086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:52:59.970157 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:52:59.970238 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:52:59.971673 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.972374 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.973550 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:52:59.973933 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:52:59.974019 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:52:59.976170 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:52:59.976175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:52:59.976178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:52:59.976205 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:52:59.978775 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:59.979360 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:59.979364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:59.979367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:59.979497 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:52:59.979801 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:52:59.979804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:52:59.979806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:52:59.980086 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:52:59.980270 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:52:59.980628 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 22:52:59.980771 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:52:59.987152 9584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:52:59.993831 9584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:52:59.998584 9773 heartbeater.cc:503] Master 127.8.204.190:37341 was elected leader, sending a full tablet report...
I20260927 22:53:00.026535 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.005931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:34727 (local address 127.0.0.1:42566)
0927 22:53:00.017637 (+ 11706us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:00.017685 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:00.017872 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:00.018702 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:00.018710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:00.018744 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:00.019115 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.019125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.020675 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.020682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.023214 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.023228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.025453 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.025460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.025682 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.025746 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 22:53:00.025874 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 22:53:00.026054 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11457,"mutex_wait_us":3,"thread_start_us":14292,"threads_started":1}
I20260927 22:53:00.027652 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.006301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.8.204.131:34727)
0927 22:53:00.017290 (+ 10989us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.017303 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.018077 (+ 774us) server_negotiation.cc:407] Connection header received
0927 22:53:00.018195 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.018204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.018497 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.018692 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:00.019321 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.020507 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.023423 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.024926 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.025054 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.025288 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.027052 (+ 1764us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.027228 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10784,"thread_start_us":10614,"threads_started":1}
I20260927 22:53:00.028632 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.006215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:33067 (local address 127.0.0.1:54100)
0927 22:53:00.020051 (+ 13836us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:00.020066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:00.020175 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:00.020982 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:00.020986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:00.021012 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:00.021485 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.021497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.023391 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.023396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.025458 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.025470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.027731 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.027737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.027923 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.027977 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:53:00.028075 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:53:00.028209 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13734,"mutex_wait_us":53,"thread_start_us":11115,"threads_started":1}
I20260927 22:53:00.029542 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.006874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.8.204.129:33067)
0927 22:53:00.018402 (+ 11528us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.018410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.020182 (+ 1772us) server_negotiation.cc:407] Connection header received
0927 22:53:00.020291 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.020298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.020524 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.020728 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:00.021682 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.023194 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.025645 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.027160 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.027257 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.027536 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.029025 (+ 1489us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.029187 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11369,"thread_start_us":11225,"threads_started":1}
I20260927 22:53:00.033349 9708 tablet_service.cc:1467] Processing CreateTablet for tablet bdcf4b0a71014168a2c10add67936b3f (DEFAULT_TABLE table=test-table [id=ba3ff3e5651f49cf880eac3042b02359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:00.033349 9975 tablet_service.cc:1467] Processing CreateTablet for tablet bdcf4b0a71014168a2c10add67936b3f (DEFAULT_TABLE table=test-table [id=ba3ff3e5651f49cf880eac3042b02359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:00.034922 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcf4b0a71014168a2c10add67936b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:00.034920 9708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcf4b0a71014168a2c10add67936b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:00.035491 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.020557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:39845 (local address 127.0.0.1:56280)
0927 22:53:00.026764 (+ 6207us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:00.026783 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:00.026883 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:00.029044 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:00.029050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:00.029079 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:00.029225 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.029232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.031167 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.031172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.032771 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.032781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.034515 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.034521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.034693 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.034781 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 22:53:00.034889 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:53:00.035142 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6120,"thread_start_us":7982,"threads_started":1}
I20260927 22:53:00.035492 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.020824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.8.204.130:39845)
0927 22:53:00.028362 (+ 7538us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.028370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.028399 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:53:00.028517 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.028526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.028808 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.029405 (+ 597us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:00.029637 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.031020 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.032926 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.034278 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.034505 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.034744 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.034943 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.035107 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7369,"thread_start_us":7239,"threads_started":1}
I20260927 22:53:00.042591 9842 tablet_service.cc:1467] Processing CreateTablet for tablet bdcf4b0a71014168a2c10add67936b3f (DEFAULT_TABLE table=test-table [id=ba3ff3e5651f49cf880eac3042b02359]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:00.044035 9842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcf4b0a71014168a2c10add67936b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:00.049909 10058 tablet_bootstrap.cc:492] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: Bootstrap starting.
I20260927 22:53:00.052356 10059 tablet_bootstrap.cc:492] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: Bootstrap starting.
I20260927 22:53:00.053844 10058 tablet_bootstrap.cc:654] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:00.055349 10058 log.cc:824] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:00.055493 10059 tablet_bootstrap.cc:654] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:00.056706 10059 log.cc:824] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:00.058328 10058 tablet_bootstrap.cc:492] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: No bootstrap required, opened a new log
I20260927 22:53:00.058640 10058 ts_tablet_manager.cc:1397] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 22:53:00.059799 10059 tablet_bootstrap.cc:492] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: No bootstrap required, opened a new log
I20260927 22:53:00.060043 10059 ts_tablet_manager.cc:1397] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 22:53:00.064881 10061 tablet_bootstrap.cc:492] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: Bootstrap starting.
I20260927 22:53:00.066866 10058 raft_consensus.cc:348] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.067323 10058 raft_consensus.cc:374] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:00.067426 10058 raft_consensus.cc:729] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4bac5732ce421fbc0ddec82f1652af, State: Initialized, Role: FOLLOWER
I20260927 22:53:00.067327 10059 raft_consensus.cc:348] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.067766 10059 raft_consensus.cc:374] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:00.067883 10059 raft_consensus.cc:729] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374ead130a26425aa499a0cb2e3f7a35, State: Initialized, Role: FOLLOWER
I20260927 22:53:00.068166 10058 consensus_queue.cc:260] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.068446 10061 tablet_bootstrap.cc:654] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:00.068454 10059 consensus_queue.cc:260] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.069726 10061 log.cc:824] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:00.070060 10058 ts_tablet_manager.cc:1428] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 22:53:00.074618 10040 heartbeater.cc:503] Master 127.8.204.190:37341 was elected leader, sending a full tablet report...
I20260927 22:53:00.075805 10059 ts_tablet_manager.cc:1428] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 22:53:00.076691 10061 tablet_bootstrap.cc:492] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: No bootstrap required, opened a new log
I20260927 22:53:00.076965 10061 ts_tablet_manager.cc:1397] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260927 22:53:00.084193 10061 raft_consensus.cc:348] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.084594 10061 raft_consensus.cc:374] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:00.084700 10061 raft_consensus.cc:729] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b127680b5d3f4f1c91b99fd9e98107f1, State: Initialized, Role: FOLLOWER
I20260927 22:53:00.085299 10061 consensus_queue.cc:260] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.086282 9907 heartbeater.cc:503] Master 127.8.204.190:37341 was elected leader, sending a full tablet report...
I20260927 22:53:00.086973 10061 ts_tablet_manager.cc:1428] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
W20260927 22:53:00.177584 10041 tablet.cc:2369] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:00.186233 9908 tablet.cc:2369] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:00.218433 9774 tablet.cc:2369] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:00.243528 10064 raft_consensus.cc:482] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:00.243813 10064 raft_consensus.cc:504] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.245483 10064 leader_election.cc:288] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe4bac5732ce421fbc0ddec82f1652af (127.8.204.129:33067), b127680b5d3f4f1c91b99fd9e98107f1 (127.8.204.130:39845)
I20260927 22:53:00.251109 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.245808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38687 (local address 127.8.204.129:33067)
0927 22:53:00.246010 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.246018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.246042 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:00.246255 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.246262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.246339 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.246443 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:00.246919 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.247958 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.249507 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.250263 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.250433 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.250507 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.250758 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.250877 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:53:00.251144 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.245662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:33067 (local address 127.8.204.131:38687)
0927 22:53:00.245859 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:00.245878 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:00.246010 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:00.246579 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:00.246582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:00.246602 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:00.246784 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.246791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.248088 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.248091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.249350 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.249361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.250413 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.250419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.250604 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.250628 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:53:00.250734 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:53:00.250858 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:53:00.251777 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.246313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:35295 (local address 127.8.204.130:39845)
0927 22:53:00.246529 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.246536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.246587 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:53:00.246818 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.246824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.246915 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.247011 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:00.247529 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.248382 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.250038 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.250793 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.250965 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.251033 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.251405 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.251538 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:53:00.252305 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.245814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:39845 (local address 127.8.204.131:35295)
0927 22:53:00.246510 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:00.246527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:00.246671 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:00.247029 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:00.247032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:00.247050 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:00.247388 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.247395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.248538 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.248543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.249891 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:00.249901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.250994 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.251000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:00.251162 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.251221 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 22:53:00.251967 (+ 746us) client_negotiation.cc:240] Negotiation successful
0927 22:53:00.252091 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":351,"threads_started":1}
I20260927 22:53:00.252177 9728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf4b0a71014168a2c10add67936b3f" candidate_uuid: "374ead130a26425aa499a0cb2e3f7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4bac5732ce421fbc0ddec82f1652af" is_pre_election: true
I20260927 22:53:00.252712 9728 raft_consensus.cc:2454] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 374ead130a26425aa499a0cb2e3f7a35 in term 0.
I20260927 22:53:00.253384 9931 leader_election.cc:302] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374ead130a26425aa499a0cb2e3f7a35, fe4bac5732ce421fbc0ddec82f1652af; no voters:
I20260927 22:53:00.253392 9862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf4b0a71014168a2c10add67936b3f" candidate_uuid: "374ead130a26425aa499a0cb2e3f7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" is_pre_election: true
I20260927 22:53:00.253741 10064 raft_consensus.cc:2781] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:00.253883 10064 raft_consensus.cc:482] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:00.253993 10064 raft_consensus.cc:3037] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:00.253953 9862 raft_consensus.cc:2454] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 374ead130a26425aa499a0cb2e3f7a35 in term 0.
I20260927 22:53:00.256417 10064 raft_consensus.cc:504] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.256989 10064 leader_election.cc:288] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [CANDIDATE]: Term 1 election: Requested vote from peers fe4bac5732ce421fbc0ddec82f1652af (127.8.204.129:33067), b127680b5d3f4f1c91b99fd9e98107f1 (127.8.204.130:39845)
I20260927 22:53:00.257365 9728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf4b0a71014168a2c10add67936b3f" candidate_uuid: "374ead130a26425aa499a0cb2e3f7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4bac5732ce421fbc0ddec82f1652af"
I20260927 22:53:00.257447 9862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcf4b0a71014168a2c10add67936b3f" candidate_uuid: "374ead130a26425aa499a0cb2e3f7a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b127680b5d3f4f1c91b99fd9e98107f1"
I20260927 22:53:00.257578 9728 raft_consensus.cc:3037] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:00.257643 9862 raft_consensus.cc:3037] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:00.260773 9862 raft_consensus.cc:2454] T bdcf4b0a71014168a2c10add67936b3f P b127680b5d3f4f1c91b99fd9e98107f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 374ead130a26425aa499a0cb2e3f7a35 in term 1.
I20260927 22:53:00.260773 9728 raft_consensus.cc:2454] T bdcf4b0a71014168a2c10add67936b3f P fe4bac5732ce421fbc0ddec82f1652af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 374ead130a26425aa499a0cb2e3f7a35 in term 1.
I20260927 22:53:00.261166 9931 leader_election.cc:302] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374ead130a26425aa499a0cb2e3f7a35, fe4bac5732ce421fbc0ddec82f1652af; no voters:
I20260927 22:53:00.261475 10064 raft_consensus.cc:2781] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:00.261909 10064 raft_consensus.cc:686] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [term 1 LEADER]: Becoming Leader. State: Replica: 374ead130a26425aa499a0cb2e3f7a35, State: Running, Role: LEADER
I20260927 22:53:00.262357 10064 consensus_queue.cc:237] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } }
I20260927 22:53:00.267067 9583 catalog_manager.cc:5495] T bdcf4b0a71014168a2c10add67936b3f P 374ead130a26425aa499a0cb2e3f7a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 374ead130a26425aa499a0cb2e3f7a35 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "374ead130a26425aa499a0cb2e3f7a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "374ead130a26425aa499a0cb2e3f7a35" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34727 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe4bac5732ce421fbc0ddec82f1652af" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 33067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b127680b5d3f4f1c91b99fd9e98107f1" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39845 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:00.340643 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.326158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.8.204.129:33067)
0927 22:53:00.326350 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.326358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.326819 (+ 461us) server_negotiation.cc:407] Connection header received
0927 22:53:00.327040 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.327045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.327108 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.327218 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:00.328778 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.329532 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.330775 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.331170 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.331319 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.335246 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:00.335259 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:00.335324 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:00.335358 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:00.338019 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.338613 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.338620 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.338622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.338743 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.339117 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.339122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.339127 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.339639 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:00.339926 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.340179 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.340363 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549580, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:00.384382 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.370828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.8.204.129:33067)
0927 22:53:00.371166 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.371172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.371501 (+ 329us) server_negotiation.cc:407] Connection header received
0927 22:53:00.371730 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.371736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.371813 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.371890 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:00.373155 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.373859 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.375100 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.375694 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.375786 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.379468 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:00.379472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:00.379476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:00.379504 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:00.382154 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.382713 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.382717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.382719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.382828 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.383156 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.383159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.383161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.383452 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:00.383630 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.383864 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.384008 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:00.429276 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.415969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.8.204.129:33067)
0927 22:53:00.416164 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.416172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.416463 (+ 291us) server_negotiation.cc:407] Connection header received
0927 22:53:00.416712 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.416716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.416778 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.416881 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:00.418167 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.418833 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.420141 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.420581 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.420713 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.424588 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:00.424593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:00.424597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:00.424628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:00.427119 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.427692 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.427696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.427698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.427805 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.428144 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.428146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.428148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.428478 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:00.428706 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.428851 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.429018 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 22:53:00.430929 9688 tablet_service.cc:2908] Rejecting scan request for tablet bdcf4b0a71014168a2c10add67936b3f: Uninitialized: clean time has not yet been initialized
I20260927 22:53:00.434042 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9645
I20260927 22:53:00.451066 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.434970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.8.204.129:33067)
0927 22:53:00.435135 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.435140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.435618 (+ 478us) server_negotiation.cc:407] Connection header received
0927 22:53:00.435742 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.435745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.435838 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.435927 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:00.437313 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.437971 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.439733 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.440323 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.440464 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.443533 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:00.443539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:00.443544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:00.443577 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:00.447308 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.448163 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.448169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.448172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.448296 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.448691 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.448696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.448698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.449045 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:00.449482 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.450094 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.450612 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:53:00.452078 9768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:00.547398 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9645
I20260927 22:53:00.564116 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9779
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:00.577595 10057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.564858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.8.204.130:39845)
0927 22:53:00.565030 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.565035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.565052 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:53:00.565151 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.565155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.565224 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.565305 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:00.566397 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.567173 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.568405 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.568771 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.568866 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.572648 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:00.572660 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:00.572674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:00.572702 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:00.575208 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.575862 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.575870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.575872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.575996 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.576383 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.576386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.576388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.576755 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:00.576966 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.577171 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.577321 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:00.578217 9902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:00.667095 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9779
I20260927 22:53:00.683596 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9912
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[9539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:00.698043 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:00.684322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.8.204.131:34727)
0927 22:53:00.684824 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:00.684828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:00.684844 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:53:00.684896 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:00.684900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:00.684957 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:00.685045 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:00.686206 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.686894 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.688160 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:00.688538 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:00.688634 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:00.692675 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:00.692688 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:00.692704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:00.692733 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:00.695612 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.696181 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.696187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.696190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.696318 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:00.696642 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:00.696645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:00.696647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:00.696943 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:00.697159 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:00.697418 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:53:00.697634 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
I20260927 22:53:00.698674 10035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:00.786623 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9912
I20260927 22:53:00.802659 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 9552
I20260927 22:53:00.803858 9614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:00.908604 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 9552
2026-09-27T22:53:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4169 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[10088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[10088](info): set up 2 sockets
Sep 27 22:53:00 dist-test-slave-40vx krb5kdc[10088](info): commencing operation
krb5kdc: starting...
W20260927 22:53:01.983407 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:02 dist-test-slave-40vx krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:02.159879 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39281
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:39281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:02.465274 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:02.465559 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:02.465746 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:02.472959 10101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:02.473104 10101 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:53:02.473176 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:02.473273 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:02.473359 10101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:02.473464 10101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:02.483119 10101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:39281
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:02.484441 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:02.485781 10101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:02.497342 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:02.497320 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:02.497700 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:02.498160 10101 server_base.cc:1034] running on GCE node
I20260927 22:53:02.498872 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:02.500228 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:02.501433 10101 hybrid_clock.cc:648] HybridClock initialized: now 1790549582501351 us; error 56 us; skew 500 ppm
I20260927 22:53:02.501700 10101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:02 dist-test-slave-40vx krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549582, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:02.504747 10101 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:02.506083 10101 webserver.cc:460] Webserver started at http://127.8.204.190:35845/ using document root <none> and password file <none>
I20260927 22:53:02.506748 10101 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:02.506835 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:02.507089 10101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:02.509995 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "e9d5187e5bbb4c65a23340155e5c1583"
format_stamp: "Formatted at 2026-09-27 22:53:02 on dist-test-slave-40vx"
server_key: "3e950651f9e27f94156f21369c8fe0a4"
server_key_iv: "8c05e0963556bf1f400e204e402c4cb0"
server_key_version: "encryptionkey@0"
I20260927 22:53:02.510722 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "e9d5187e5bbb4c65a23340155e5c1583"
format_stamp: "Formatted at 2026-09-27 22:53:02 on dist-test-slave-40vx"
server_key: "3e950651f9e27f94156f21369c8fe0a4"
server_key_iv: "8c05e0963556bf1f400e204e402c4cb0"
server_key_version: "encryptionkey@0"
I20260927 22:53:02.515643 10101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 22:53:02.518901 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:02.520334 10101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:02.520457 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "e9d5187e5bbb4c65a23340155e5c1583"
format_stamp: "Formatted at 2026-09-27 22:53:02 on dist-test-slave-40vx"
server_key: "3e950651f9e27f94156f21369c8fe0a4"
server_key_iv: "8c05e0963556bf1f400e204e402c4cb0"
server_key_version: "encryptionkey@0"
I20260927 22:53:02.520624 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:02.533641 10101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:02.537397 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:02.537613 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:02.558486 10101 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:39281
I20260927 22:53:02.558514 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:39281 every 8 connection(s)
I20260927 22:53:02.559831 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:02.563431 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:02.565379 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10101
I20260927 22:53:02.565600 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:02.566231 9010 external_mini_cluster.cc:1428] Setting key 14bf2c7bd3c855be3f450b1cb6a5ca8e
I20260927 22:53:02.575634 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: Bootstrap starting.
I20260927 22:53:02.579010 10178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:02.580008 10178 log.cc:824] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:53:02 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:02.582994 10178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: No bootstrap required, opened a new log
I20260927 22:53:02.590288 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:02.568255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.8.204.190:39281)
0927 22:53:02.569070 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:02.569085 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:02.569133 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:53:02.571468 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:02.571495 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:02.571973 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:02.572409 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:02.573564 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:02.575433 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:02.576616 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:02.577694 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:02.577871 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:02.581927 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:02.581938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:02.581957 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:02.581997 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:02.585905 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:02.586538 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:02.586546 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:02.586554 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:02.586758 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:02.587065 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:02.587071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:02.587074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:02.587858 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:02.588150 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:02.588442 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:53:02.588894 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":349,"threads_started":1}
I20260927 22:53:02.593477 10178 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d5187e5bbb4c65a23340155e5c1583" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39281 } }
I20260927 22:53:02.593786 10178 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:02.593860 10178 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d5187e5bbb4c65a23340155e5c1583, State: Initialized, Role: FOLLOWER
I20260927 22:53:02.594364 10178 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d5187e5bbb4c65a23340155e5c1583" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39281 } }
I20260927 22:53:02.594560 10178 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:02.594653 10178 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:02.594766 10178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:02.596829 10178 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d5187e5bbb4c65a23340155e5c1583" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39281 } }
I20260927 22:53:02.597247 10178 leader_election.cc:302] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9d5187e5bbb4c65a23340155e5c1583; no voters:
I20260927 22:53:02.597842 10178 leader_election.cc:288] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:02.597958 10183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:02.598941 10183 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [term 1 LEADER]: Becoming Leader. State: Replica: e9d5187e5bbb4c65a23340155e5c1583, State: Running, Role: LEADER
I20260927 22:53:02.599517 10183 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d5187e5bbb4c65a23340155e5c1583" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39281 } }
I20260927 22:53:02.600019 10178 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:02.602044 10185 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9d5187e5bbb4c65a23340155e5c1583. Latest consensus state: current_term: 1 leader_uuid: "e9d5187e5bbb4c65a23340155e5c1583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d5187e5bbb4c65a23340155e5c1583" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39281 } } }
I20260927 22:53:02.602402 10185 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:02.603821 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9d5187e5bbb4c65a23340155e5c1583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9d5187e5bbb4c65a23340155e5c1583" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39281 } } }
I20260927 22:53:02.604239 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:02.604887 10188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:02.610314 10188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:02.611966 10200 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:53:02.619505 10188 catalog_manager.cc:1347] Generated new cluster ID: 98310b6d727848fc85e313e01913f0ea
I20260927 22:53:02.619640 10188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:02.647125 10188 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:02.648599 10188 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:02.655500 10188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: Generated new TSK 0
I20260927 22:53:02.656311 10188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:02.707741 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:39281
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:03.033672 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:03.033923 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:03.034015 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:03.040977 10204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:03.041070 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:03.041154 10204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:03.041229 10204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:03.050540 10204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:39281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:03.051834 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:03.053104 10204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:03.065848 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:03.065891 10210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:03.065954 10212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:03.066838 10204 server_base.cc:1034] running on GCE node
I20260927 22:53:03.067517 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:03.068392 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:03.069605 10204 hybrid_clock.cc:648] HybridClock initialized: now 1790549583069533 us; error 54 us; skew 500 ppm
I20260927 22:53:03.069892 10204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:03 dist-test-slave-40vx krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:03.073062 10204 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:03.074426 10204 webserver.cc:460] Webserver started at http://127.8.204.129:38317/ using document root <none> and password file <none>
I20260927 22:53:03.075109 10204 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:03.075232 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:03.075556 10204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:03.078356 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b"
format_stamp: "Formatted at 2026-09-27 22:53:03 on dist-test-slave-40vx"
server_key: "e22e83186fc73613db2f0a608fc86693"
server_key_iv: "f98a8644e684bf18de39b459b8fcf411"
server_key_version: "encryptionkey@0"
I20260927 22:53:03.079125 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b"
format_stamp: "Formatted at 2026-09-27 22:53:03 on dist-test-slave-40vx"
server_key: "e22e83186fc73613db2f0a608fc86693"
server_key_iv: "f98a8644e684bf18de39b459b8fcf411"
server_key_version: "encryptionkey@0"
I20260927 22:53:03.084273 10204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260927 22:53:03.087657 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:03.088982 10204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:53:03.089146 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b"
format_stamp: "Formatted at 2026-09-27 22:53:03 on dist-test-slave-40vx"
server_key: "e22e83186fc73613db2f0a608fc86693"
server_key_iv: "f98a8644e684bf18de39b459b8fcf411"
server_key_version: "encryptionkey@0"
I20260927 22:53:03.089349 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:03.109697 10204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:03.113816 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:03.114073 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:03.116582 10204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:03.116691 10204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:03.116842 10204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:03.116912 10204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:03 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:03.139935 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.116989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:44187 (local address 127.8.204.190:39281)
0927 22:53:03.117516 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:03.117522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:03.118732 (+ 1210us) server_negotiation.cc:407] Connection header received
0927 22:53:03.120879 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:03.120888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:03.120995 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:03.121130 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:03.123314 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.124405 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.127015 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.127530 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.127686 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.132508 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:03.132518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:03.132525 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:03.132571 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:03.136405 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.137287 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.137292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.137296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.137414 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.137953 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.137961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.137964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.138352 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:03.138484 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:03.139381 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 22:53:03.139581 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
I20260927 22:53:03.140879 10229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.117689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.8.204.129:44187)
0927 22:53:03.118555 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:03.118601 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:03.120613 (+ 2012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:03.121403 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:03.121412 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:03.122176 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:03.123044 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.123072 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.124583 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.124594 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.126839 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.126854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.127802 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.127816 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.128195 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.128934 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:03.128980 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:03.132216 (+ 3236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:03.136535 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.136541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.136553 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.137107 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.137534 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.137541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.137547 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.137824 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.138475 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:03.138482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:03.138933 (+ 451us) client_negotiation.cc:769] Sending connection context
0927 22:53:03.139270 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 22:53:03.139767 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":379,"threads_started":1}
I20260927 22:53:03.159605 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.146365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:59141 (local address 127.8.204.190:39281)
0927 22:53:03.146575 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:03.146583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:03.147113 (+ 530us) server_negotiation.cc:407] Connection header received
0927 22:53:03.147247 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:03.147255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:03.147358 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:03.147476 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:03.148716 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.149393 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.150634 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.151015 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.151148 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.153636 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:03.153644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:03.153664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:03.153702 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:03.156512 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.157480 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.157489 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.157495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.157618 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.158096 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.158104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.158108 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.158540 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:03.158734 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:03.159053 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:53:03.159256 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:53:03.159720 10286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.146352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.8.204.129:59141)
0927 22:53:03.147026 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:03.147042 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:03.147139 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:03.147521 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:03.147528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:03.148098 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:03.148536 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.148549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.149520 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.149525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.150496 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.150507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.151189 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.151195 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.151391 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.152049 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:03.152081 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:03.153424 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:03.156663 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.156671 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.156677 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.157340 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.157751 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.157755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.157759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.157950 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.158666 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:03.158669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:03.158810 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 22:53:03.158953 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:53:03.159259 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":380,"threads_started":1}
I20260927 22:53:03.173280 10204 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:36419
I20260927 22:53:03.173302 10339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:36419 every 8 connection(s)
I20260927 22:53:03.174481 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:03.175882 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10204
I20260927 22:53:03.176100 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:03.176609 10340 heartbeater.cc:347] Connected to a master server at 127.8.204.190:39281
I20260927 22:53:03.176721 9010 external_mini_cluster.cc:1428] Setting key c804a93245ed1c39f105204aa5e24cb9
I20260927 22:53:03.176795 10340 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:03.177410 10340 heartbeater.cc:511] Master 127.8.204.190:39281 requested a full tablet report, sending...
I20260927 22:53:03.179330 10133 ts_manager.cc:194] Registered new tserver with Master: 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129:36419)
I20260927 22:53:03.181484 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:44187
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:03.218673 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:39281
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:03.531663 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:03.531980 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:03.532125 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:03.539124 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:03.539270 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:03.539446 10344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:03.539601 10344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:03.549371 10344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:39281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:03.550704 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:03.552110 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:03.567688 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:03.567632 10350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:03.568523 10352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:03.568794 10344 server_base.cc:1034] running on GCE node
I20260927 22:53:03.569490 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:03.570444 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:03.571705 10344 hybrid_clock.cc:648] HybridClock initialized: now 1790549583571622 us; error 69 us; skew 500 ppm
I20260927 22:53:03.572021 10344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:03 dist-test-slave-40vx krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:03.575194 10344 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:03.576649 10344 webserver.cc:460] Webserver started at http://127.8.204.130:33361/ using document root <none> and password file <none>
I20260927 22:53:03.577344 10344 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:03.577467 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:03.577768 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:03.580677 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "d62c048d4af5492099027f370e519675"
format_stamp: "Formatted at 2026-09-27 22:53:03 on dist-test-slave-40vx"
server_key: "510189c50bed96ae9ad43543f30f1847"
server_key_iv: "d0ff9d82bd3c0f1323d69ed1ed614244"
server_key_version: "encryptionkey@0"
I20260927 22:53:03.581490 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "d62c048d4af5492099027f370e519675"
format_stamp: "Formatted at 2026-09-27 22:53:03 on dist-test-slave-40vx"
server_key: "510189c50bed96ae9ad43543f30f1847"
server_key_iv: "d0ff9d82bd3c0f1323d69ed1ed614244"
server_key_version: "encryptionkey@0"
I20260927 22:53:03.586714 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:53:03.590181 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:03.591542 10344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:03.591701 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "d62c048d4af5492099027f370e519675"
format_stamp: "Formatted at 2026-09-27 22:53:03 on dist-test-slave-40vx"
server_key: "510189c50bed96ae9ad43543f30f1847"
server_key_iv: "d0ff9d82bd3c0f1323d69ed1ed614244"
server_key_version: "encryptionkey@0"
I20260927 22:53:03.591907 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:03.610546 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:03.614816 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:03.615115 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:03.618388 10344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:03.618494 10344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:03.618620 10344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:03.618700 10344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:03 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549583, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:03.641428 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.619089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:47363 (local address 127.8.204.190:39281)
0927 22:53:03.619292 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:03.619299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:03.620649 (+ 1350us) server_negotiation.cc:407] Connection header received
0927 22:53:03.622735 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:03.622741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:03.622824 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:03.622940 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:03.625021 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.626064 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.628665 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.629233 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.629375 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.634418 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:03.634428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:03.634432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:03.634463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:03.637894 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.638633 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.638642 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.638648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.638837 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.639241 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.639249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.639255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.639705 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:03.639880 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:03.640639 (+ 759us) server_negotiation.cc:299] Negotiation successful
0927 22:53:03.640933 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:53:03.642187 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.619624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.8.204.130:47363)
0927 22:53:03.620481 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:03.620534 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:03.622429 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:03.623178 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:03.623185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:03.623980 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:03.624848 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.624868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.626284 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.626294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.628439 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.628453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.629469 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.629479 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.629835 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.630703 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:03.630732 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:03.634200 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:03.638045 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.638055 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.638073 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.638487 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.638947 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.638950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.638952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.639129 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.639850 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:03.639858 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:03.640183 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 22:53:03.640524 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 22:53:03.640893 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":370,"threads_started":1}
I20260927 22:53:03.663260 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.648853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:38653 (local address 127.8.204.190:39281)
0927 22:53:03.649077 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:03.649085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:03.649428 (+ 343us) server_negotiation.cc:407] Connection header received
0927 22:53:03.649672 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:03.649678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:03.649750 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:03.649854 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:03.650839 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.652336 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.653653 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.654547 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.654688 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.656992 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:03.657001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:03.657008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:03.657053 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:03.660360 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.661302 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.661308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.661311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.661440 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:03.661810 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:03.661815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:03.661819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:03.662123 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:03.662271 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:03.662675 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:53:03.662890 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:53:03.663417 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:03.648430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.8.204.130:38653)
0927 22:53:03.649300 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:03.649327 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:03.649519 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:03.650000 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:03.650003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:03.650412 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:03.650686 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.650693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.652486 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.652490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.653515 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:03.653526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:03.654746 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:03.654751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:03.654988 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:03.655771 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:03.655795 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:03.656794 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:03.660523 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.660528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.660532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.661168 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.661552 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:03.661560 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:03.661564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:03.661690 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:03.662241 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:03.662246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:03.662431 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 22:53:03.662591 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:53:03.662910 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":529,"threads_started":1}
I20260927 22:53:03.673048 10344 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:42581
I20260927 22:53:03.673092 10478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:42581 every 8 connection(s)
I20260927 22:53:03.674222 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:03.675060 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10344
I20260927 22:53:03.675256 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:03.675915 9010 external_mini_cluster.cc:1428] Setting key 7b2ba3ef21c7bc84b0fe1f69d925326d
I20260927 22:53:03.676597 10479 heartbeater.cc:347] Connected to a master server at 127.8.204.190:39281
I20260927 22:53:03.676791 10479 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:03.677552 10479 heartbeater.cc:511] Master 127.8.204.190:39281 requested a full tablet report, sending...
I20260927 22:53:03.678834 10133 ts_manager.cc:194] Registered new tserver with Master: d62c048d4af5492099027f370e519675 (127.8.204.130:42581)
I20260927 22:53:03.679694 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:47363
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:03.714555 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:39281
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:04.031958 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:04.032249 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:04.032413 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:04.039433 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:04.039579 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:04.039711 10483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:04.039835 10483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:04.049678 10483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:39281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:04.050855 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:04.052300 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:04.064697 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:04.064672 10491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:04.064755 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:04.065676 10483 server_base.cc:1034] running on GCE node
I20260927 22:53:04.066288 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:04.067190 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:04.068471 10483 hybrid_clock.cc:648] HybridClock initialized: now 1790549584068375 us; error 32 us; skew 500 ppm
I20260927 22:53:04.068804 10483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:04 dist-test-slave-40vx krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:04.071898 10483 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:04.073436 10483 webserver.cc:460] Webserver started at http://127.8.204.131:32969/ using document root <none> and password file <none>
I20260927 22:53:04.074149 10483 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:04.074276 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:04.074604 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:04.077613 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "57b54926999b4dccb91cc7e445bb1de5"
format_stamp: "Formatted at 2026-09-27 22:53:04 on dist-test-slave-40vx"
server_key: "5a6ac4eb133205c8c422dd567e6df647"
server_key_iv: "9e3df43c84d4f9bfc7efd0b90bbb9d4c"
server_key_version: "encryptionkey@0"
I20260927 22:53:04.078425 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "57b54926999b4dccb91cc7e445bb1de5"
format_stamp: "Formatted at 2026-09-27 22:53:04 on dist-test-slave-40vx"
server_key: "5a6ac4eb133205c8c422dd567e6df647"
server_key_iv: "9e3df43c84d4f9bfc7efd0b90bbb9d4c"
server_key_version: "encryptionkey@0"
I20260927 22:53:04.083819 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 22:53:04.087318 10498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:04.088639 10483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:04.088822 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "57b54926999b4dccb91cc7e445bb1de5"
format_stamp: "Formatted at 2026-09-27 22:53:04 on dist-test-slave-40vx"
server_key: "5a6ac4eb133205c8c422dd567e6df647"
server_key_iv: "9e3df43c84d4f9bfc7efd0b90bbb9d4c"
server_key_version: "encryptionkey@0"
I20260927 22:53:04.089066 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:04.118327 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:04.122212 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:04.122462 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:04.125123 10483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:04.125237 10483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:04.125396 10483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:04.125479 10483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:04 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:04.152223 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.126012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:43883 (local address 127.8.204.190:39281)
0927 22:53:04.126187 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.126193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.127808 (+ 1615us) server_negotiation.cc:407] Connection header received
0927 22:53:04.130328 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.130334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.130433 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.130563 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:04.133195 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.134344 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.137141 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.137821 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.137989 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.143738 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:04.143745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:04.143751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:04.143813 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:04.147730 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.148809 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.148815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.148819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.148970 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.149610 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.149616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.149620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.150062 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:04.150276 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.151442 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.151743 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:53:04.154227 10507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.126610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.8.204.131:43883)
0927 22:53:04.127623 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.127680 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.129915 (+ 2235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.130895 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.130909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.131909 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:04.132974 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.133007 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.134602 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.134613 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.136945 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.136970 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.138092 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.138103 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.138476 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.139443 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:04.139476 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:04.143422 (+ 3946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:04.147922 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:04.147936 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:04.147955 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:04.148658 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:04.149104 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:04.149109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:04.149119 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:04.149486 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:04.150217 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:04.150230 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:04.150854 (+ 624us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.151245 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.152045 (+ 800us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":425,"threads_started":1}
I20260927 22:53:04.167567 10483 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:44353
I20260927 22:53:04.167600 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:44353 every 8 connection(s)
I20260927 22:53:04.168982 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:04.170390 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10483
I20260927 22:53:04.170573 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:04.171054 9010 external_mini_cluster.cc:1428] Setting key 7040eec139182fe2ee08f77c5447dc6d
I20260927 22:53:04.171058 10618 heartbeater.cc:347] Connected to a master server at 127.8.204.190:39281
I20260927 22:53:04.172031 10618 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:04.172995 10618 heartbeater.cc:511] Master 127.8.204.190:39281 requested a full tablet report, sending...
I20260927 22:53:04.174741 10133 ts_manager.cc:194] Registered new tserver with Master: 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.175822 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:43883
I20260927 22:53:04.179540 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.161980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:34759 (local address 127.8.204.190:39281)
0927 22:53:04.162201 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.162210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.163845 (+ 1635us) server_negotiation.cc:407] Connection header received
0927 22:53:04.164045 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.164052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.164146 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.164249 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:04.165609 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.166481 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.168589 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.168991 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.169131 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.172928 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:04.172934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:04.172939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:04.172976 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:04.176702 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.177572 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.177577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.177582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.177715 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.178098 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.178103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.178107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.178411 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:04.178547 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.179066 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.179235 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:53:04.179778 10611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.162036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.8.204.131:34759)
0927 22:53:04.163733 (+ 1697us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.163753 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.163913 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.164339 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.164347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.165043 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:04.165433 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.165447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.166630 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.166638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.168448 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.168462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.169192 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.169198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.169380 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.171763 (+ 2383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:04.171786 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:04.172707 (+ 921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:04.176867 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:04.176873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:04.176877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:04.177439 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:04.177839 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:04.177843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:04.177847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:04.177979 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:04.178628 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:04.178632 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:04.178816 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.178954 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.179265 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1550,"thread_start_us":1405,"threads_started":1}
I20260927 22:53:04.184926 10340 heartbeater.cc:503] Master 127.8.204.190:39281 was elected leader, sending a full tablet report...
I20260927 22:53:04.186628 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:53:04 dist-test-slave-40vx krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:53:04 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:04.230183 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.216891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.8.204.190:39281)
0927 22:53:04.217116 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.217124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.217392 (+ 268us) server_negotiation.cc:407] Connection header received
0927 22:53:04.217610 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.217620 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.217682 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.217786 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:04.219088 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.219884 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.221134 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.221556 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.221685 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.225660 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:04.225665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:04.225669 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:04.225702 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:04.228080 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.228656 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.228660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.228662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.228779 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.229066 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.229069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.229071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.229311 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:04.229482 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.229737 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.229889 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:53:04.234016 10133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:04.240607 10133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:04.263850 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.253199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.8.204.130:42581)
0927 22:53:04.254162 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.254175 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.255756 (+ 1581us) server_negotiation.cc:407] Connection header received
0927 22:53:04.256126 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.256138 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.256437 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.256627 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.257761 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.259173 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.261637 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.262637 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.262781 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.262982 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.263337 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.263534 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":658,"threads_started":1}
I20260927 22:53:04.264415 10632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.252659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42581 (local address 127.0.0.1:55878)
0927 22:53:04.255614 (+ 2955us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.255636 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.255877 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.256983 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.256993 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.257056 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.257577 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.257591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.259357 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.259361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.261490 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.261500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.262850 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.262856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.263062 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.263095 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.263902 (+ 807us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.264102 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2766,"thread_start_us":2606,"threads_started":1}
I20260927 22:53:04.266095 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.252688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.8.204.129:36419)
0927 22:53:04.257578 (+ 4890us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.257586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.257619 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:53:04.257737 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.257747 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.257971 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.259287 (+ 1316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.259481 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.260944 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.263585 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.265051 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.265194 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.265392 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.265629 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.265786 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4702,"thread_start_us":4572,"threads_started":1}
I20260927 22:53:04.266093 10631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.252505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:36419 (local address 127.0.0.1:34688)
0927 22:53:04.254872 (+ 2367us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.254920 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.255136 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.258181 (+ 3045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.258186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.258212 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.258562 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.258575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.261070 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.261074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.263382 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.263398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.265261 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.265267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.265436 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.265469 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.265571 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.265700 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2042,"mutex_wait_us":49,"thread_start_us":1845,"threads_started":1}
I20260927 22:53:04.266656 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.256105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.8.204.131:44353)
0927 22:53:04.256899 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.256911 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.258121 (+ 1210us) server_negotiation.cc:407] Connection header received
0927 22:53:04.258244 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.258254 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.258489 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.258690 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.259807 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.260798 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.263837 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.265217 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.265325 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.265508 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.266192 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.266351 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":397,"threads_started":1}
I20260927 22:53:04.267119 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.255629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:44353 (local address 127.0.0.1:53726)
0927 22:53:04.257990 (+ 2361us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.258006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.258126 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.259039 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.259044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.259067 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.259554 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.259567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.260932 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.260940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.263675 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.263686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.265660 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.265666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.265825 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.265855 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.265948 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.266077 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2261,"mutex_wait_us":60,"thread_start_us":2170,"threads_started":1}
I20260927 22:53:04.268172 10407 tablet_service.cc:1467] Processing CreateTablet for tablet c16134a2edaf4020bc82145ddf4e7180 (DEFAULT_TABLE table=test-table [id=150a1ba5b4c949d5ac50f869c439fcd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:04.269405 10407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16134a2edaf4020bc82145ddf4e7180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.269400 10270 tablet_service.cc:1467] Processing CreateTablet for tablet c16134a2edaf4020bc82145ddf4e7180 (DEFAULT_TABLE table=test-table [id=150a1ba5b4c949d5ac50f869c439fcd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:04.270372 10270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16134a2edaf4020bc82145ddf4e7180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.274367 10548 tablet_service.cc:1467] Processing CreateTablet for tablet c16134a2edaf4020bc82145ddf4e7180 (DEFAULT_TABLE table=test-table [id=150a1ba5b4c949d5ac50f869c439fcd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:04.275820 10548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c16134a2edaf4020bc82145ddf4e7180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.280355 10637 tablet_bootstrap.cc:492] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Bootstrap starting.
I20260927 22:53:04.283705 10638 tablet_bootstrap.cc:492] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Bootstrap starting.
I20260927 22:53:04.284536 10637 tablet_bootstrap.cc:654] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.285772 10637 log.cc:824] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:04.286999 10638 tablet_bootstrap.cc:654] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.288225 10638 log.cc:824] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:04.288718 10640 tablet_bootstrap.cc:492] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Bootstrap starting.
I20260927 22:53:04.288900 10637 tablet_bootstrap.cc:492] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: No bootstrap required, opened a new log
I20260927 22:53:04.289215 10637 ts_tablet_manager.cc:1397] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260927 22:53:04.291874 10638 tablet_bootstrap.cc:492] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: No bootstrap required, opened a new log
I20260927 22:53:04.292135 10640 tablet_bootstrap.cc:654] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.292140 10638 ts_tablet_manager.cc:1397] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:53:04.293651 10640 log.cc:824] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:04.296533 10640 tablet_bootstrap.cc:492] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: No bootstrap required, opened a new log
I20260927 22:53:04.296779 10640 ts_tablet_manager.cc:1397] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:53:04.297681 10637 raft_consensus.cc:348] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.298007 10637 raft_consensus.cc:374] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.298101 10637 raft_consensus.cc:729] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62c048d4af5492099027f370e519675, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.298660 10637 consensus_queue.cc:260] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.299711 10479 heartbeater.cc:503] Master 127.8.204.190:39281 was elected leader, sending a full tablet report...
I20260927 22:53:04.300269 10637 ts_tablet_manager.cc:1428] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260927 22:53:04.302525 10638 raft_consensus.cc:348] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.302912 10638 raft_consensus.cc:374] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.303027 10638 raft_consensus.cc:729] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5a6c70d3e4faaabb2e1331cf3a21b, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.303632 10638 consensus_queue.cc:260] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.308704 10638 ts_tablet_manager.cc:1428] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260927 22:53:04.310999 10640 raft_consensus.cc:348] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.311434 10640 raft_consensus.cc:374] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.311514 10640 raft_consensus.cc:729] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b54926999b4dccb91cc7e445bb1de5, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.312036 10640 consensus_queue.cc:260] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.313020 10618 heartbeater.cc:503] Master 127.8.204.190:39281 was elected leader, sending a full tablet report...
I20260927 22:53:04.313653 10640 ts_tablet_manager.cc:1428] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260927 22:53:04.421160 10619 tablet.cc:2369] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:04.426259 10480 tablet.cc:2369] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:04.426735 10341 tablet.cc:2369] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:04.451174 10644 raft_consensus.cc:482] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:04.451551 10644 raft_consensus.cc:504] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.453183 10644 leader_election.cc:288] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d62c048d4af5492099027f370e519675 (127.8.204.130:42581), 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.460366 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.453661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:57199 (local address 127.8.204.130:42581)
0927 22:53:04.453895 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.453903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.454262 (+ 359us) server_negotiation.cc:407] Connection header received
0927 22:53:04.454507 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.454514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.454601 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.454701 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.455484 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.456404 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.458338 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.459234 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.459434 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.459509 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.459977 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.460111 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:53:04.460470 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.453528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42581 (local address 127.8.204.129:57199)
0927 22:53:04.454157 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.454177 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.454368 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.454819 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.454825 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.454855 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.455348 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.455360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.456525 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.456531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.458200 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.458211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.459696 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.459701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.459825 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.459847 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.459922 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.460035 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1,"spinlock_wait_cycles":25472,"thread_start_us":311,"threads_started":1}
I20260927 22:53:04.460457 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.454535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:37591 (local address 127.8.204.131:44353)
0927 22:53:04.454786 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.454792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.455004 (+ 212us) server_negotiation.cc:407] Connection header received
0927 22:53:04.455142 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.455149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.455232 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.455360 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.455939 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.456745 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.458672 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.459515 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.459710 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.459782 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.460089 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.460226 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 22:53:04.460525 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.454198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:44353 (local address 127.8.204.129:37591)
0927 22:53:04.454908 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.454921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.455027 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.455365 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.455368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.455385 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.455789 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.455803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.456901 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.456909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.458525 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.458541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.459741 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.459748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.459925 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.459958 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.460050 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.460172 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":405,"threads_started":1}
I20260927 22:53:04.461488 10434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16134a2edaf4020bc82145ddf4e7180" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62c048d4af5492099027f370e519675" is_pre_election: true
I20260927 22:53:04.461488 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16134a2edaf4020bc82145ddf4e7180" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b54926999b4dccb91cc7e445bb1de5" is_pre_election: true
I20260927 22:53:04.461943 10568 raft_consensus.cc:2454] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 0.
I20260927 22:53:04.461951 10434 raft_consensus.cc:2454] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 0.
I20260927 22:53:04.462520 10223 leader_election.cc:302] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5a6c70d3e4faaabb2e1331cf3a21b, 57b54926999b4dccb91cc7e445bb1de5; no voters:
I20260927 22:53:04.462908 10644 raft_consensus.cc:2781] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:04.463056 10644 raft_consensus.cc:482] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:04.463164 10644 raft_consensus.cc:3037] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.465279 10644 raft_consensus.cc:504] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.465831 10644 leader_election.cc:288] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 election: Requested vote from peers d62c048d4af5492099027f370e519675 (127.8.204.130:42581), 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.466236 10434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16134a2edaf4020bc82145ddf4e7180" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62c048d4af5492099027f370e519675"
I20260927 22:53:04.466306 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c16134a2edaf4020bc82145ddf4e7180" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b54926999b4dccb91cc7e445bb1de5"
I20260927 22:53:04.466423 10434 raft_consensus.cc:3037] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.466529 10568 raft_consensus.cc:3037] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.468572 10434 raft_consensus.cc:2454] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 1.
I20260927 22:53:04.468705 10568 raft_consensus.cc:2454] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 1.
I20260927 22:53:04.468961 10223 leader_election.cc:302] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5a6c70d3e4faaabb2e1331cf3a21b, d62c048d4af5492099027f370e519675; no voters:
I20260927 22:53:04.469277 10644 raft_consensus.cc:2781] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:04.469741 10644 raft_consensus.cc:686] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 LEADER]: Becoming Leader. State: Replica: 19b5a6c70d3e4faaabb2e1331cf3a21b, State: Running, Role: LEADER
I20260927 22:53:04.470266 10644 consensus_queue.cc:237] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.475749 10133 catalog_manager.cc:5495] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b reported cstate change: term changed from 0 to 1, leader changed from <none> to 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129). New cstate: current_term: 1 leader_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } health_report { overall_health: UNKNOWN } } }
I20260927 22:53:04.503449 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.498221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.8.204.129:36419)
0927 22:53:04.498393 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.498398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.498481 (+ 83us) server_negotiation.cc:407] Connection header received
0927 22:53:04.498736 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.498739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.498818 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.498927 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:04.499224 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.500006 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.501392 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.501782 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.501878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.502344 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.502842 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.502948 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.503127 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.503189 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:53:04.511384 10568 raft_consensus.cc:1261] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Refusing update from remote peer 19b5a6c70d3e4faaabb2e1331cf3a21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:04.511384 10434 raft_consensus.cc:1261] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Refusing update from remote peer 19b5a6c70d3e4faaabb2e1331cf3a21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:04.512320 10648 consensus_queue.cc:1035] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:04.512696 10644 consensus_queue.cc:1035] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:04.525010 10651 mvcc.cc:205] Tried to move back new op lower bound from 7334091098148765696 to 7334091097993441280. Current Snapshot: MvccSnapshot[applied={T|T < 7334091098148765696}]
I20260927 22:53:04.525856 10652 mvcc.cc:205] Tried to move back new op lower bound from 7334091098148765696 to 7334091097993441280. Current Snapshot: MvccSnapshot[applied={T|T < 7334091098148765696}]
I20260927 22:53:04.526938 10653 mvcc.cc:205] Tried to move back new op lower bound from 7334091098148765696 to 7334091097993441280. Current Snapshot: MvccSnapshot[applied={T|T < 7334091098148765696}]
I20260927 22:53:04.543418 10132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:04.543730 10132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:04.547247 10132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: Sending DeleteTablet for 3 replicas of tablet c16134a2edaf4020bc82145ddf4e7180
I20260927 22:53:04.548380 10270 tablet_service.cc:1514] Processing DeleteTablet for tablet c16134a2edaf4020bc82145ddf4e7180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:04 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34688
I20260927 22:53:04.548488 10407 tablet_service.cc:1514] Processing DeleteTablet for tablet c16134a2edaf4020bc82145ddf4e7180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:04 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:55878
I20260927 22:53:04.549382 10548 tablet_service.cc:1514] Processing DeleteTablet for tablet c16134a2edaf4020bc82145ddf4e7180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:04 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:53726
I20260927 22:53:04.549827 10132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:04.550511 10132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:04.558127 10269 tablet_service.cc:1467] Processing CreateTablet for tablet fe0adce48a3541c4aed16b1565474a15 (DEFAULT_TABLE table=test-table [id=e664a5d35b294409b009d3fe680717ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:04.558476 10409 tablet_service.cc:1467] Processing CreateTablet for tablet fe0adce48a3541c4aed16b1565474a15 (DEFAULT_TABLE table=test-table [id=e664a5d35b294409b009d3fe680717ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:04.558727 10269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0adce48a3541c4aed16b1565474a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.559008 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0adce48a3541c4aed16b1565474a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.564035 10547 tablet_service.cc:1467] Processing CreateTablet for tablet fe0adce48a3541c4aed16b1565474a15 (DEFAULT_TABLE table=test-table [id=e664a5d35b294409b009d3fe680717ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:04.564594 10547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe0adce48a3541c4aed16b1565474a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.566300 10638 tablet_bootstrap.cc:492] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Bootstrap starting.
I20260927 22:53:04.568085 10638 tablet_bootstrap.cc:654] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.579761 10638 tablet_bootstrap.cc:492] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: No bootstrap required, opened a new log
I20260927 22:53:04.580029 10638 ts_tablet_manager.cc:1397] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.001s
I20260927 22:53:04.581003 10640 tablet_bootstrap.cc:492] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Bootstrap starting.
I20260927 22:53:04.581123 10638 raft_consensus.cc:348] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.581423 10638 raft_consensus.cc:374] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.581588 10638 raft_consensus.cc:729] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5a6c70d3e4faaabb2e1331cf3a21b, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.581984 10638 consensus_queue.cc:260] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.582677 10638 ts_tablet_manager.cc:1428] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:04.582794 10640 tablet_bootstrap.cc:654] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.585881 10662 tablet_replica.cc:330] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: stopping tablet replica
I20260927 22:53:04.586457 10662 raft_consensus.cc:2229] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:04.587340 10637 tablet_bootstrap.cc:492] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Bootstrap starting.
I20260927 22:53:04.587884 10661 tablet_replica.cc:330] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: stopping tablet replica
I20260927 22:53:04.587325 10662 raft_consensus.cc:2258] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:04.588704 10661 raft_consensus.cc:2229] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:04.590659 10662 ts_tablet_manager.cc:1905] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:04.589100 10637 tablet_bootstrap.cc:654] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.595817 10661 raft_consensus.cc:2258] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:04.598325 10661 ts_tablet_manager.cc:1905] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:04.598703 10640 tablet_bootstrap.cc:492] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: No bootstrap required, opened a new log
I20260927 22:53:04.598865 10640 ts_tablet_manager.cc:1397] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.007s
I20260927 22:53:04.599776 10640 raft_consensus.cc:348] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.600003 10640 raft_consensus.cc:374] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.600107 10640 raft_consensus.cc:729] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b54926999b4dccb91cc7e445bb1de5, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.600378 10640 consensus_queue.cc:260] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.603236 10640 ts_tablet_manager.cc:1428] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 22:53:04.606371 10637 tablet_bootstrap.cc:492] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: No bootstrap required, opened a new log
I20260927 22:53:04.606555 10637 ts_tablet_manager.cc:1397] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.000s
I20260927 22:53:04.607026 10663 tablet_replica.cc:330] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: stopping tablet replica
I20260927 22:53:04.607021 10662 ts_tablet_manager.cc:1918] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:04.607306 10662 log.cc:1196] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/c16134a2edaf4020bc82145ddf4e7180
I20260927 22:53:04.607522 10663 raft_consensus.cc:2229] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:04.607618 10637 raft_consensus.cc:348] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.607901 10637 raft_consensus.cc:374] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.608023 10637 raft_consensus.cc:729] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62c048d4af5492099027f370e519675, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.608036 10663 raft_consensus.cc:2258] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:04.608213 10662 ts_tablet_manager.cc:1939] T c16134a2edaf4020bc82145ddf4e7180 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Deleting consensus metadata
I20260927 22:53:04.608314 10637 consensus_queue.cc:260] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.610147 10663 ts_tablet_manager.cc:1905] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:04.610193 10117 catalog_manager.cc:4841] TS 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129:36419): tablet c16134a2edaf4020bc82145ddf4e7180 (table test-table [id=150a1ba5b4c949d5ac50f869c439fcd6]) successfully deleted
I20260927 22:53:04.610965 10637 ts_tablet_manager.cc:1428] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:53:04.616259 10661 ts_tablet_manager.cc:1918] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:04.616428 10661 log.cc:1196] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/c16134a2edaf4020bc82145ddf4e7180
I20260927 22:53:04.616626 10663 ts_tablet_manager.cc:1918] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:04.616777 10663 log.cc:1196] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/c16134a2edaf4020bc82145ddf4e7180
I20260927 22:53:04.617055 10661 ts_tablet_manager.cc:1939] T c16134a2edaf4020bc82145ddf4e7180 P d62c048d4af5492099027f370e519675: Deleting consensus metadata
I20260927 22:53:04.617224 10663 ts_tablet_manager.cc:1939] T c16134a2edaf4020bc82145ddf4e7180 P 57b54926999b4dccb91cc7e445bb1de5: Deleting consensus metadata
I20260927 22:53:04.618959 10120 catalog_manager.cc:4841] TS d62c048d4af5492099027f370e519675 (127.8.204.130:42581): tablet c16134a2edaf4020bc82145ddf4e7180 (table test-table [id=150a1ba5b4c949d5ac50f869c439fcd6]) successfully deleted
I20260927 22:53:04.619217 10120 catalog_manager.cc:4841] TS 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353): tablet c16134a2edaf4020bc82145ddf4e7180 (table test-table [id=150a1ba5b4c949d5ac50f869c439fcd6]) successfully deleted
I20260927 22:53:04.634371 10656 raft_consensus.cc:482] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:04.634567 10656 raft_consensus.cc:504] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.635357 10656 leader_election.cc:288] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d62c048d4af5492099027f370e519675 (127.8.204.130:42581), 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.635797 10433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0adce48a3541c4aed16b1565474a15" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62c048d4af5492099027f370e519675" is_pre_election: true
I20260927 22:53:04.635897 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0adce48a3541c4aed16b1565474a15" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b54926999b4dccb91cc7e445bb1de5" is_pre_election: true
I20260927 22:53:04.636061 10433 raft_consensus.cc:2454] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 0.
I20260927 22:53:04.636168 10568 raft_consensus.cc:2454] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 0.
I20260927 22:53:04.636541 10223 leader_election.cc:302] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5a6c70d3e4faaabb2e1331cf3a21b, 57b54926999b4dccb91cc7e445bb1de5; no voters:
I20260927 22:53:04.636848 10656 raft_consensus.cc:2781] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:04.636987 10656 raft_consensus.cc:482] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:04.637102 10656 raft_consensus.cc:3037] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.639182 10656 raft_consensus.cc:504] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.639720 10656 leader_election.cc:288] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 election: Requested vote from peers d62c048d4af5492099027f370e519675 (127.8.204.130:42581), 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.640209 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0adce48a3541c4aed16b1565474a15" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b54926999b4dccb91cc7e445bb1de5"
I20260927 22:53:04.640213 10433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe0adce48a3541c4aed16b1565474a15" candidate_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d62c048d4af5492099027f370e519675"
I20260927 22:53:04.640386 10568 raft_consensus.cc:3037] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.640407 10433 raft_consensus.cc:3037] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.642303 10568 raft_consensus.cc:2454] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 1.
I20260927 22:53:04.642683 10223 leader_election.cc:302] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5a6c70d3e4faaabb2e1331cf3a21b, 57b54926999b4dccb91cc7e445bb1de5; no voters:
I20260927 22:53:04.642949 10656 raft_consensus.cc:2781] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:04.643134 10656 raft_consensus.cc:686] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 LEADER]: Becoming Leader. State: Replica: 19b5a6c70d3e4faaabb2e1331cf3a21b, State: Running, Role: LEADER
I20260927 22:53:04.643468 10656 consensus_queue.cc:237] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.644197 10433 raft_consensus.cc:2454] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19b5a6c70d3e4faaabb2e1331cf3a21b in term 1.
I20260927 22:53:04.646672 10133 catalog_manager.cc:5495] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b reported cstate change: term changed from 0 to 1, leader changed from <none> to 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129). New cstate: current_term: 1 leader_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:53:04 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549582, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:04.701889 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.685714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.8.204.190:39281)
0927 22:53:04.685878 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.685884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.686507 (+ 623us) server_negotiation.cc:407] Connection header received
0927 22:53:04.686679 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.686686 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.686779 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.686893 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:04.688276 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.689543 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.691423 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.692153 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.692288 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.696534 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:04.696538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:04.696541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:04.696568 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:04.698932 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.699881 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.699886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.699893 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.700004 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.700434 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.700440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.700445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.700695 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:04.700853 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.701255 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.701552 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":53}
I20260927 22:53:04.701906 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.685804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39281 (local address 127.0.0.1:44990)
0927 22:53:04.686385 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.686412 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.686550 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.686920 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.686925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.687790 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:04.688114 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.688125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.689696 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.689701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.691240 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.691253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.692328 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.692335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.692496 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.693328 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:04.693351 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:04.696384 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:04.699060 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:04.699072 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:04.699080 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:04.699749 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:04.700112 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:04.700118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:04.700122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:04.700296 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:04.700830 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:04.700836 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:04.701008 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.701158 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.701434 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":325,"threads_started":1}
I20260927 22:53:04.707571 10133 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:44990:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:53:04.708870 10133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:04.715420 10270 tablet_service.cc:1467] Processing CreateTablet for tablet 164ebac415b9487e81dd583cd88b5388 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de467f72d26249eb9dae2925468c9b8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:04.716035 10270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164ebac415b9487e81dd583cd88b5388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.716013 10548 tablet_service.cc:1467] Processing CreateTablet for tablet 164ebac415b9487e81dd583cd88b5388 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de467f72d26249eb9dae2925468c9b8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:04.716225 10409 tablet_service.cc:1467] Processing CreateTablet for tablet 164ebac415b9487e81dd583cd88b5388 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=de467f72d26249eb9dae2925468c9b8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:04.716526 10548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164ebac415b9487e81dd583cd88b5388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.716679 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 164ebac415b9487e81dd583cd88b5388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:04.723491 10637 tablet_bootstrap.cc:492] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: Bootstrap starting.
I20260927 22:53:04.724285 10638 tablet_bootstrap.cc:492] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Bootstrap starting.
I20260927 22:53:04.725150 10637 tablet_bootstrap.cc:654] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.725684 10638 tablet_bootstrap.cc:654] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.726706 10640 tablet_bootstrap.cc:492] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: Bootstrap starting.
I20260927 22:53:04.728117 10637 tablet_bootstrap.cc:492] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: No bootstrap required, opened a new log
I20260927 22:53:04.728310 10640 tablet_bootstrap.cc:654] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:04.728437 10637 ts_tablet_manager.cc:1397] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 22:53:04.729449 10637 raft_consensus.cc:348] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.729765 10637 raft_consensus.cc:374] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.729913 10637 raft_consensus.cc:729] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62c048d4af5492099027f370e519675, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.730333 10637 consensus_queue.cc:260] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.730917 10643 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.731334 10643 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.731901 10637 ts_tablet_manager.cc:1428] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 22:53:04.734970 10643 raft_consensus.cc:482] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:04.735198 10643 raft_consensus.cc:504] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.737064 10643 leader_election.cc:288] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129:36419), 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.742179 10638 tablet_bootstrap.cc:492] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: No bootstrap required, opened a new log
I20260927 22:53:04.742353 10638 ts_tablet_manager.cc:1397] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20260927 22:53:04.743294 10638 raft_consensus.cc:348] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.743536 10638 raft_consensus.cc:374] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.743636 10638 raft_consensus.cc:729] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19b5a6c70d3e4faaabb2e1331cf3a21b, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.743979 10638 consensus_queue.cc:260] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.744400 10656 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.744694 10656 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.747305 10638 ts_tablet_manager.cc:1428] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260927 22:53:04.748226 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.738261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:55867 (local address 127.8.204.129:36419)
0927 22:53:04.738472 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.738481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.739882 (+ 1401us) server_negotiation.cc:407] Connection header received
0927 22:53:04.740229 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.740237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.740355 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.740439 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.741415 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.742624 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.745648 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.746875 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.747008 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.747114 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.747768 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.747896 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:53:04.749248 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.737240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:36419 (local address 127.8.204.130:55867)
0927 22:53:04.739766 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.739791 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.740010 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.740687 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.740695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.740728 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.741197 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.741210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.742922 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.742927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.745479 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.745494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.747332 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.747339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.747584 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.747619 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.748735 (+ 1116us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.748872 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2377,"thread_start_us":3927,"threads_started":1}
I20260927 22:53:04.750196 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164ebac415b9487e81dd583cd88b5388" candidate_uuid: "d62c048d4af5492099027f370e519675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" is_pre_election: true
I20260927 22:53:04.750636 10295 raft_consensus.cc:2454] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62c048d4af5492099027f370e519675 in term 0.
I20260927 22:53:04.751245 10360 leader_election.cc:302] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5a6c70d3e4faaabb2e1331cf3a21b, d62c048d4af5492099027f370e519675; no voters:
I20260927 22:53:04.751682 10643 raft_consensus.cc:2781] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:04.751794 10643 raft_consensus.cc:482] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:04.751888 10643 raft_consensus.cc:3037] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.752264 10640 tablet_bootstrap.cc:492] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: No bootstrap required, opened a new log
I20260927 22:53:04.752475 10640 ts_tablet_manager.cc:1397] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.008s
I20260927 22:53:04.753216 10640 raft_consensus.cc:348] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.753449 10640 raft_consensus.cc:374] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:04.753528 10640 raft_consensus.cc:729] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b54926999b4dccb91cc7e445bb1de5, State: Initialized, Role: FOLLOWER
I20260927 22:53:04.753765 10640 consensus_queue.cc:260] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.754220 10645 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.754503 10645 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.754945 10640 ts_tablet_manager.cc:1428] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:53:04.754788 10643 raft_consensus.cc:504] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.755492 10643 leader_election.cc:288] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [CANDIDATE]: Term 1 election: Requested vote from peers 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129:36419), 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353)
I20260927 22:53:04.756115 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.737801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:41141 (local address 127.8.204.131:44353)
0927 22:53:04.738036 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.738045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.741553 (+ 3508us) server_negotiation.cc:407] Connection header received
0927 22:53:04.742451 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.742457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.742532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.742591 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:04.744297 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.745479 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.750364 (+ 4885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.751675 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.751795 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.751864 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.755699 (+ 3835us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.755839 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:53:04.756407 10295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164ebac415b9487e81dd583cd88b5388" candidate_uuid: "d62c048d4af5492099027f370e519675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b"
I20260927 22:53:04.756639 10295 raft_consensus.cc:3037] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.759338 10295 raft_consensus.cc:2454] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62c048d4af5492099027f370e519675 in term 1.
I20260927 22:53:04.759788 10360 leader_election.cc:302] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19b5a6c70d3e4faaabb2e1331cf3a21b, d62c048d4af5492099027f370e519675; no voters:
I20260927 22:53:04.760043 10643 raft_consensus.cc:2781] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:04.760119 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.737775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:44353 (local address 127.8.204.130:41141)
0927 22:53:04.741456 (+ 3681us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.741475 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.742130 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.742787 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.742792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.742816 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:04.743876 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.743889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.746681 (+ 2792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.746686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.749842 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.749857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.755302 (+ 5445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.755308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.755492 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.755529 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.759360 (+ 3831us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.759514 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3565,"mutex_wait_us":3,"thread_start_us":1688,"threads_started":1}
I20260927 22:53:04.760537 10643 raft_consensus.cc:686] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [term 1 LEADER]: Becoming Leader. State: Replica: d62c048d4af5492099027f370e519675, State: Running, Role: LEADER
I20260927 22:53:04.760794 10568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164ebac415b9487e81dd583cd88b5388" candidate_uuid: "d62c048d4af5492099027f370e519675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b54926999b4dccb91cc7e445bb1de5" is_pre_election: true
I20260927 22:53:04.761022 10568 raft_consensus.cc:2454] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d62c048d4af5492099027f370e519675 in term 0.
I20260927 22:53:04.760929 10643 consensus_queue.cc:237] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } }
I20260927 22:53:04.761327 10567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164ebac415b9487e81dd583cd88b5388" candidate_uuid: "d62c048d4af5492099027f370e519675" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57b54926999b4dccb91cc7e445bb1de5"
I20260927 22:53:04.761545 10567 raft_consensus.cc:3037] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:04.763130 10679 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: TxnStatusTablet state changed. Reason: New leader d62c048d4af5492099027f370e519675. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.763504 10679 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:04.764357 10567 raft_consensus.cc:2454] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d62c048d4af5492099027f370e519675 in term 1.
I20260927 22:53:04.764760 10133 catalog_manager.cc:5495] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 reported cstate change: term changed from 0 to 1, leader changed from <none> to d62c048d4af5492099027f370e519675 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } health_report { overall_health: UNKNOWN } } }
I20260927 22:53:04.771696 10681 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:53:04.771941 10681 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:53:04.793520 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.786342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42581 (local address 127.0.0.1:55882)
0927 22:53:04.786603 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.786624 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.786758 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.787222 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.787227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.787240 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:04.787465 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.787476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.788927 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.788933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.790465 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.790479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.791522 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.791528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.791744 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.791808 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.792921 (+ 1113us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.792939 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.793019 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.793171 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 22:53:04.794229 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.786636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.8.204.130:42581)
0927 22:53:04.786794 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.786800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.786820 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:53:04.786954 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.786959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.787044 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.787186 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:04.787617 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.788793 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.790625 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.791353 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.791494 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.792053 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.792685 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.792868 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.793840 (+ 972us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.793930 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:53:04.798382 10567 raft_consensus.cc:1261] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Refusing update from remote peer d62c048d4af5492099027f370e519675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:04.798610 10295 raft_consensus.cc:1261] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Refusing update from remote peer d62c048d4af5492099027f370e519675: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:04.799120 10643 consensus_queue.cc:1035] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:04.799786 10643 consensus_queue.cc:1035] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:04.806523 10645 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: TxnStatusTablet state changed. Reason: New leader d62c048d4af5492099027f370e519675. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.806807 10645 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.813024 10656 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: TxnStatusTablet state changed. Reason: New leader d62c048d4af5492099027f370e519675. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.813347 10656 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.814568 10679 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.814817 10679 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:04.816705 10643 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.817028 10643 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P d62c048d4af5492099027f370e519675: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:04.818922 10645 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.819173 10645 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P 57b54926999b4dccb91cc7e445bb1de5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.821554 10656 tablet_replica.cc:439] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d62c048d4af5492099027f370e519675" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19b5a6c70d3e4faaabb2e1331cf3a21b" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36419 } } peers { permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 } } peers { permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 } } }
I20260927 22:53:04.821834 10656 tablet_replica.cc:442] T 164ebac415b9487e81dd583cd88b5388 P 19b5a6c70d3e4faaabb2e1331cf3a21b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:04.844426 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.836089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:35355 (local address 127.8.204.130:42581)
0927 22:53:04.836297 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.836304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.836998 (+ 694us) server_negotiation.cc:407] Connection header received
0927 22:53:04.837195 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.837203 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.837293 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.837392 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:04.837999 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.839207 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.841441 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.842185 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.842323 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.843458 (+ 1135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.843606 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.843714 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.843990 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.844076 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:53:04.844561 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.836126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42581 (local address 127.8.204.129:35355)
0927 22:53:04.836872 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.836893 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.837052 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.837421 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.837426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.837439 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:04.837823 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.837835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.839376 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.839382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.841280 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.841296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.842463 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.842470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.842673 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.842743 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.843786 (+ 1043us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.843805 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.843970 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.844141 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":362,"threads_started":1}
I20260927 22:53:04.855352 10567 raft_consensus.cc:1261] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Refusing update from remote peer 19b5a6c70d3e4faaabb2e1331cf3a21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:04.856086 10644 consensus_queue.cc:1035] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "57b54926999b4dccb91cc7e445bb1de5" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:04.857118 10433 raft_consensus.cc:1261] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Refusing update from remote peer 19b5a6c70d3e4faaabb2e1331cf3a21b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:04.858585 10644 consensus_queue.cc:1035] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d62c048d4af5492099027f370e519675" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:04.908852 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.900842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:36419 (local address 127.8.204.130:58887)
0927 22:53:04.901726 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:04.901747 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:04.901957 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:04.902339 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:04.902344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:04.902357 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:04.902760 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.902774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.904416 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.904423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.906322 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:04.906336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.907428 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.907435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:04.907622 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.907700 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.908170 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.908181 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:53:04.908276 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:53:04.908433 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":540,"threads_started":1}
I20260927 22:53:04.909750 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.900848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:58887 (local address 127.8.204.129:36419)
0927 22:53:04.901091 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.901101 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.901830 (+ 729us) server_negotiation.cc:407] Connection header received
0927 22:53:04.902122 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.902127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.902210 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.902301 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:04.902981 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.904247 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.906479 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.907224 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.907384 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.907892 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:04.908049 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:04.908145 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.908309 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.908392 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:53:04.949172 10132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:04.949426 10132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:04.952927 10132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9d5187e5bbb4c65a23340155e5c1583: Sending DeleteTablet for 3 replicas of tablet fe0adce48a3541c4aed16b1565474a15
I20260927 22:53:04.953635 10270 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0adce48a3541c4aed16b1565474a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:04 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34688
I20260927 22:53:04.954178 10662 tablet_replica.cc:330] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: stopping tablet replica
I20260927 22:53:04.954142 10548 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0adce48a3541c4aed16b1565474a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:04 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:53726
I20260927 22:53:04.954412 10663 tablet_replica.cc:330] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: stopping tablet replica
I20260927 22:53:04.954399 10662 raft_consensus.cc:2229] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:04.954603 10663 raft_consensus.cc:2229] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:04.954751 10662 raft_consensus.cc:2258] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:04.954774 10663 raft_consensus.cc:2258] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:04.955246 10409 tablet_service.cc:1514] Processing DeleteTablet for tablet fe0adce48a3541c4aed16b1565474a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:04 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:55878
I20260927 22:53:04.955549 10661 tablet_replica.cc:330] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: stopping tablet replica
I20260927 22:53:04.955745 10661 raft_consensus.cc:2229] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:04.955929 10661 raft_consensus.cc:2258] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:04.956096 10663 ts_tablet_manager.cc:1905] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:04.957183 10662 ts_tablet_manager.cc:1905] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:04.960503 10661 ts_tablet_manager.cc:1905] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:04.964769 10663 ts_tablet_manager.cc:1918] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:04.964933 10663 log.cc:1196] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/fe0adce48a3541c4aed16b1565474a15
I20260927 22:53:04.965380 10663 ts_tablet_manager.cc:1939] T fe0adce48a3541c4aed16b1565474a15 P 57b54926999b4dccb91cc7e445bb1de5: Deleting consensus metadata
I20260927 22:53:04.966394 10120 catalog_manager.cc:4841] TS 57b54926999b4dccb91cc7e445bb1de5 (127.8.204.131:44353): tablet fe0adce48a3541c4aed16b1565474a15 (table test-table [id=e664a5d35b294409b009d3fe680717ec]) successfully deleted
I20260927 22:53:04.971534 10661 ts_tablet_manager.cc:1918] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:04.971695 10661 log.cc:1196] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/fe0adce48a3541c4aed16b1565474a15
I20260927 22:53:04.972218 10661 ts_tablet_manager.cc:1939] T fe0adce48a3541c4aed16b1565474a15 P d62c048d4af5492099027f370e519675: Deleting consensus metadata
I20260927 22:53:04.973268 10120 catalog_manager.cc:4841] TS d62c048d4af5492099027f370e519675 (127.8.204.130:42581): tablet fe0adce48a3541c4aed16b1565474a15 (table test-table [id=e664a5d35b294409b009d3fe680717ec]) successfully deleted
I20260927 22:53:04.978228 10662 ts_tablet_manager.cc:1918] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:04.978473 10662 log.cc:1196] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/fe0adce48a3541c4aed16b1565474a15
I20260927 22:53:04.979049 10662 ts_tablet_manager.cc:1939] T fe0adce48a3541c4aed16b1565474a15 P 19b5a6c70d3e4faaabb2e1331cf3a21b: Deleting consensus metadata
I20260927 22:53:04.980281 10117 catalog_manager.cc:4841] TS 19b5a6c70d3e4faaabb2e1331cf3a21b (127.8.204.129:36419): tablet fe0adce48a3541c4aed16b1565474a15 (table test-table [id=e664a5d35b294409b009d3fe680717ec]) successfully deleted
Sep 27 22:53:04 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:04.996395 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:04.969024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.8.204.129:36419)
0927 22:53:04.969197 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:04.969204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:04.972736 (+ 3532us) server_negotiation.cc:407] Connection header received
0927 22:53:04.973010 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:04.973020 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:04.973096 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:04.973170 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:04.975263 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.976526 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.978779 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:04.979489 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:04.979616 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:04.988475 (+ 8859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:04.988493 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:04.988513 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:04.988558 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:04.992789 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.993763 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.993773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.993779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.993933 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:04.994428 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:04.994433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:04.994447 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:04.994935 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:04.995338 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:04.995682 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 22:53:04.995917 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 22:53:04.997131 10335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34696
I20260927 22:53:05.023625 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:05.004814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.8.204.190:39281)
0927 22:53:05.005016 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:05.005022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:05.007792 (+ 2770us) server_negotiation.cc:407] Connection header received
0927 22:53:05.007987 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:05.007994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:05.008090 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:05.008200 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:05.009998 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.011085 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.012899 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.013542 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.013664 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:05.016800 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:05.016807 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:05.016812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:05.016850 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:05.020403 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.021207 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.021214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.021219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.021364 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.021789 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.021794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.021798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.022162 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:05.022429 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:05.022747 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:53:05.022986 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:53:05.024246 10132 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45002
I20260927 22:53:05.026175 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10204
I20260927 22:53:05.044000 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:05.027698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.8.204.129:36419)
0927 22:53:05.027885 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:05.027891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:05.028315 (+ 424us) server_negotiation.cc:407] Connection header received
0927 22:53:05.028578 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:05.028585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:05.028666 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:05.028734 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:05.030347 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.031507 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.033218 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.033877 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.034024 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:05.036886 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:05.036894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:05.036905 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:05.036954 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:05.040808 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.041645 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.041652 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.041662 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.041822 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.042260 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.042265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.042269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.042719 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:05.043005 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:05.043342 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:53:05.043640 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:53:05.045385 10335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34710
W20260927 22:53:05.045951 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:05.046147 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10204
I20260927 22:53:05.067363 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10344
Sep 27 22:53:05 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:05.083518 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:05.068050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.8.204.130:42581)
0927 22:53:05.068265 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:05.068272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:05.068310 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:53:05.068521 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:05.068526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:05.068595 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:05.068702 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:05.069766 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.070552 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.071841 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.072308 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.072446 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:05.076569 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:05.076582 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:05.076599 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:05.076647 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:05.080707 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.081345 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.081355 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.081359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.081514 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.081891 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.081897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.081901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.082391 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:05.082725 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:05.082872 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 22:53:05.083114 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:53:05.084218 10474 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55898
W20260927 22:53:05.084828 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10344: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:05.085011 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10344
I20260927 22:53:05.103914 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10483
Sep 27 22:53:05 dist-test-slave-40vx krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549584, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:05.117980 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:05.104693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.8.204.131:44353)
0927 22:53:05.104861 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:05.104866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:05.104883 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:53:05.104952 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:05.104956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:05.105030 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:05.105109 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:05.106292 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.107109 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.108529 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:05.108943 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:05.109066 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:05.113101 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:05.113112 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:05.113128 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:05.113156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:05.115787 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.116353 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.116359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.116361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.116468 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:05.116762 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:05.116765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:05.116767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:05.117081 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:05.117280 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:05.117500 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 22:53:05.117661 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:53:05.118669 10613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53728
W20260927 22:53:05.119239 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:05.119400 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10483
I20260927 22:53:05.135990 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10101
I20260927 22:53:05.137099 10173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:05.249790 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10101
2026-09-27T22:53:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4342 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:05 dist-test-slave-40vx krb5kdc[10705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:05 dist-test-slave-40vx krb5kdc[10705](info): set up 2 sockets
Sep 27 22:53:05 dist-test-slave-40vx krb5kdc[10705](info): commencing operation
krb5kdc: starting...
W20260927 22:53:06.327576 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:06 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:06.503777 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45711
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:45711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:06.813400 10718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:06.813725 10718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:06.813870 10718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:06.821154 10718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:06.821295 10718 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:53:06.821383 10718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:06.821502 10718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:06.821583 10718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:06.821671 10718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:06.831640 10718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:45711
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:06.832935 10718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:06.834311 10718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:06.845932 10723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:06.845856 10724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:06.845880 10726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:06.846920 10718 server_base.cc:1034] running on GCE node
I20260927 22:53:06.847741 10718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:06.849193 10718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:06.850452 10718 hybrid_clock.cc:648] HybridClock initialized: now 1790549586850396 us; error 48 us; skew 500 ppm
I20260927 22:53:06.850797 10718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:06 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:06.853842 10718 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:06.855235 10718 webserver.cc:460] Webserver started at http://127.8.204.190:41199/ using document root <none> and password file <none>
I20260927 22:53:06.855952 10718 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:06.856065 10718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:06.856357 10718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:06.859606 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "360e06bb6de24c449b5cebdfc4a6f113"
format_stamp: "Formatted at 2026-09-27 22:53:06 on dist-test-slave-40vx"
server_key: "d1e37877bbf61bc750f1aa2b751f32da"
server_key_iv: "60fdd6a65e29d6703250793dfd7eb82c"
server_key_version: "encryptionkey@0"
I20260927 22:53:06.860318 10718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "360e06bb6de24c449b5cebdfc4a6f113"
format_stamp: "Formatted at 2026-09-27 22:53:06 on dist-test-slave-40vx"
server_key: "d1e37877bbf61bc750f1aa2b751f32da"
server_key_iv: "60fdd6a65e29d6703250793dfd7eb82c"
server_key_version: "encryptionkey@0"
I20260927 22:53:06.865083 10718 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:53:06.868295 10733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:06.869516 10718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:53:06.869693 10718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "360e06bb6de24c449b5cebdfc4a6f113"
format_stamp: "Formatted at 2026-09-27 22:53:06 on dist-test-slave-40vx"
server_key: "d1e37877bbf61bc750f1aa2b751f32da"
server_key_iv: "60fdd6a65e29d6703250793dfd7eb82c"
server_key_version: "encryptionkey@0"
I20260927 22:53:06.869886 10718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:06.884622 10718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:06.888299 10718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:06.888552 10718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:06.909832 10718 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:45711
I20260927 22:53:06.909854 10794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:45711 every 8 connection(s)
I20260927 22:53:06.911141 10718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:06.914803 10795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:06.919824 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10718
I20260927 22:53:06.920020 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:06.920764 9010 external_mini_cluster.cc:1428] Setting key fbc9525d91dc31ed7adb80015f3518f0
I20260927 22:53:06.925716 10795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Bootstrap starting.
I20260927 22:53:06.929370 10795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:06.930536 10795 log.cc:824] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:06.933667 10795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: No bootstrap required, opened a new log
Sep 27 22:53:06 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:06.941581 10795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e06bb6de24c449b5cebdfc4a6f113" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45711 } }
I20260927 22:53:06.941939 10795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:06.942068 10795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e06bb6de24c449b5cebdfc4a6f113, State: Initialized, Role: FOLLOWER
I20260927 22:53:06.942649 10795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e06bb6de24c449b5cebdfc4a6f113" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45711 } }
I20260927 22:53:06.942888 10795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:06.943034 10795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:06.943205 10795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:06.943591 10798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:06.922714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42380 (local address 127.8.204.190:45711)
0927 22:53:06.923561 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:06.923575 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:06.923617 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:53:06.925263 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:06.925284 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:06.925651 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:06.925992 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:06.927702 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:06.929086 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:06.930820 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:06.931908 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:06.932038 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:06.936463 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:06.936470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:06.936483 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:06.936520 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:06.939472 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:06.940154 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:06.940163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:06.940169 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:06.940338 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:06.940690 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:06.940696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:06.940699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:06.941180 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:06.941468 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:06.941878 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 22:53:06.942449 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":380,"threads_started":1}
I20260927 22:53:06.945501 10795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e06bb6de24c449b5cebdfc4a6f113" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45711 } }
I20260927 22:53:06.945956 10795 leader_election.cc:302] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 360e06bb6de24c449b5cebdfc4a6f113; no voters:
I20260927 22:53:06.946523 10795 leader_election.cc:288] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:06.946632 10800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:06.947321 10800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [term 1 LEADER]: Becoming Leader. State: Replica: 360e06bb6de24c449b5cebdfc4a6f113, State: Running, Role: LEADER
I20260927 22:53:06.947780 10800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e06bb6de24c449b5cebdfc4a6f113" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45711 } }
I20260927 22:53:06.948458 10795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:06.950407 10802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360e06bb6de24c449b5cebdfc4a6f113. Latest consensus state: current_term: 1 leader_uuid: "360e06bb6de24c449b5cebdfc4a6f113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e06bb6de24c449b5cebdfc4a6f113" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45711 } } }
I20260927 22:53:06.950721 10802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:06.950176 10801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360e06bb6de24c449b5cebdfc4a6f113" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e06bb6de24c449b5cebdfc4a6f113" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45711 } } }
I20260927 22:53:06.951310 10801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:06.952344 10805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:06.957538 10805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:06.966099 10805 catalog_manager.cc:1347] Generated new cluster ID: ef229e2251d243fe93fe51c8dab98f77
I20260927 22:53:06.966243 10805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:07.015125 10805 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:07.017038 10805 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:07.040251 10805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Generated new TSK 0
I20260927 22:53:07.041124 10805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:07.098479 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45711
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:07.422716 10821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:07.422994 10821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:07.423161 10821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:07.430374 10821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:07.430477 10821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:07.430637 10821 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:53:07.430713 10821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:07.430848 10821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:07.440546 10821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:45711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:07.441888 10821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:07.443161 10821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:07.455933 10827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:07.455976 10826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:07.455897 10829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:07.457041 10821 server_base.cc:1034] running on GCE node
I20260927 22:53:07.457526 10821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:07.458348 10821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:07.459602 10821 hybrid_clock.cc:648] HybridClock initialized: now 1790549587459567 us; error 34 us; skew 500 ppm
I20260927 22:53:07.459851 10821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:07 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:07.462618 10821 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:07.463939 10821 webserver.cc:460] Webserver started at http://127.8.204.129:45215/ using document root <none> and password file <none>
I20260927 22:53:07.464571 10821 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:07.464701 10821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:07.464988 10821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:07.467746 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "9e9987f4c9e0403d824db44b0c859525"
format_stamp: "Formatted at 2026-09-27 22:53:07 on dist-test-slave-40vx"
server_key: "b64457e4681b68b4610534fb7bf91f19"
server_key_iv: "1773fcae57eb23a8d1419b532ee2aaf5"
server_key_version: "encryptionkey@0"
I20260927 22:53:07.468454 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "9e9987f4c9e0403d824db44b0c859525"
format_stamp: "Formatted at 2026-09-27 22:53:07 on dist-test-slave-40vx"
server_key: "b64457e4681b68b4610534fb7bf91f19"
server_key_iv: "1773fcae57eb23a8d1419b532ee2aaf5"
server_key_version: "encryptionkey@0"
I20260927 22:53:07.473258 10821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:53:07.476475 10836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:07.477658 10821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:07.477810 10821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "9e9987f4c9e0403d824db44b0c859525"
format_stamp: "Formatted at 2026-09-27 22:53:07 on dist-test-slave-40vx"
server_key: "b64457e4681b68b4610534fb7bf91f19"
server_key_iv: "1773fcae57eb23a8d1419b532ee2aaf5"
server_key_version: "encryptionkey@0"
I20260927 22:53:07.477983 10821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:07.492240 10821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:07.495874 10821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:07.496138 10821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:07.498492 10821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:07.498622 10821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:07.498759 10821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:07.498850 10821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:07 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:07.523010 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:07.499207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:56605 (local address 127.8.204.190:45711)
0927 22:53:07.499721 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:07.499727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:07.500831 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 22:53:07.502850 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:07.502856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:07.502958 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:07.503076 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:07.505671 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.506596 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.509860 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.510449 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.510621 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:07.515671 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:07.515679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:07.515686 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:07.515750 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:07.518896 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:07.519769 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:07.519776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:07.519781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:07.519934 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:07.520539 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:07.520547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:07.520552 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:07.521035 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:07.521242 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:07.522189 (+ 947us) server_negotiation.cc:299] Negotiation successful
0927 22:53:07.522473 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":304,"threads_started":1}
I20260927 22:53:07.523784 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:07.499815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.8.204.129:56605)
0927 22:53:07.500657 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:07.500709 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:07.502582 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:07.503524 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:07.503536 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:07.504464 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:07.505454 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:07.505490 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.506809 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.506818 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:07.509671 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:07.509690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.510769 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.510776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:07.511173 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:07.512067 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:07.512109 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:07.515422 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:07.519035 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:07.519044 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:07.519065 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:07.519578 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:07.520053 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:07.520058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:07.520062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:07.520374 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:07.521191 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:07.521197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:07.521610 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 22:53:07.522007 (+ 397us) client_negotiation.cc:240] Negotiation successful
0927 22:53:07.522566 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":376,"threads_started":1}
I20260927 22:53:07.546679 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:07.529874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:57887 (local address 127.8.204.190:45711)
0927 22:53:07.530121 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:07.530129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:07.531955 (+ 1826us) server_negotiation.cc:407] Connection header received
0927 22:53:07.532111 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:07.532119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:07.532219 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:07.532348 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:07.533196 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.534614 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.536263 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.536848 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.537003 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:07.539517 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:07.539528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:07.539536 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:07.539583 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:07.543274 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:07.544144 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:07.544152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:07.544158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:07.544315 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:07.544753 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:07.544757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:07.544761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:07.545158 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:07.545352 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:07.545926 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 22:53:07.546201 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:53:07.546813 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:07.529368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.8.204.129:57887)
0927 22:53:07.531855 (+ 2487us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:07.531874 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:07.531989 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:07.532354 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:07.532357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:07.532793 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:07.533066 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:07.533076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.534751 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.534754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:07.536075 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:07.536086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:07.537116 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:07.537126 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:07.537259 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:07.537887 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:07.537902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:07.539255 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:07.543443 (+ 4188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:07.543451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:07.543459 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:07.544013 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:07.544440 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:07.544447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:07.544452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:07.544628 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:07.545287 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:07.545294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:07.545528 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 22:53:07.545735 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 22:53:07.546171 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2329,"thread_start_us":2115,"threads_started":1}
I20260927 22:53:07.548767 10821 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:43495
I20260927 22:53:07.548838 10956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:43495 every 8 connection(s)
I20260927 22:53:07.550339 10821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:07.552315 10957 heartbeater.cc:347] Connected to a master server at 127.8.204.190:45711
I20260927 22:53:07.552522 10957 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:07.553113 10957 heartbeater.cc:511] Master 127.8.204.190:45711 requested a full tablet report, sending...
I20260927 22:53:07.554965 10750 ts_manager.cc:194] Registered new tserver with Master: 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495)
I20260927 22:53:07.556386 10750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:56605
I20260927 22:53:07.557845 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10821
I20260927 22:53:07.558045 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:07.558802 9010 external_mini_cluster.cc:1428] Setting key 9c6e7dce4231429e4b2f1ed151d33533
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:07.598537 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45711
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:07.915007 10961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:07.915316 10961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:07.915503 10961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:07.922478 10961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:07.922569 10961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:07.922734 10961 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:53:07.922806 10961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:07.922917 10961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:07.932592 10961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:45711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:07.933733 10961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:07.934930 10961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:07.946556 10966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:07.947304 10967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:07.948290 10969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:07.948884 10961 server_base.cc:1034] running on GCE node
I20260927 22:53:07.949435 10961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:07.950287 10961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:07.951483 10961 hybrid_clock.cc:648] HybridClock initialized: now 1790549587951417 us; error 55 us; skew 500 ppm
I20260927 22:53:07.951771 10961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:07 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:07.954754 10961 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:07.956180 10961 webserver.cc:460] Webserver started at http://127.8.204.130:41503/ using document root <none> and password file <none>
I20260927 22:53:07.956827 10961 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:07.956965 10961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:07.957249 10961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:07.960175 10961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "feb7399116844192a4a38bf6d102cbf6"
format_stamp: "Formatted at 2026-09-27 22:53:07 on dist-test-slave-40vx"
server_key: "54de4e8280a7e981b5c97aed4e1e6a0f"
server_key_iv: "a3418ac4b72ffdeb5ad8e26d3f9d3d0a"
server_key_version: "encryptionkey@0"
I20260927 22:53:07.960911 10961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "feb7399116844192a4a38bf6d102cbf6"
format_stamp: "Formatted at 2026-09-27 22:53:07 on dist-test-slave-40vx"
server_key: "54de4e8280a7e981b5c97aed4e1e6a0f"
server_key_iv: "a3418ac4b72ffdeb5ad8e26d3f9d3d0a"
server_key_version: "encryptionkey@0"
I20260927 22:53:07.966136 10961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 22:53:07.970845 10976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:07.972757 10961 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:53:07.972930 10961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "feb7399116844192a4a38bf6d102cbf6"
format_stamp: "Formatted at 2026-09-27 22:53:07 on dist-test-slave-40vx"
server_key: "54de4e8280a7e981b5c97aed4e1e6a0f"
server_key_iv: "a3418ac4b72ffdeb5ad8e26d3f9d3d0a"
server_key_version: "encryptionkey@0"
I20260927 22:53:07.973177 10961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:07.991822 10961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:07.995478 10961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:07.995715 10961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:07.998111 10961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:07.998214 10961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:07.998355 10961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:07.998446 10961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:08 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549587, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:08.026888 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:07.998805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:56975 (local address 127.8.204.190:45711)
0927 22:53:07.998990 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:07.998996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.000383 (+ 1387us) server_negotiation.cc:407] Connection header received
0927 22:53:08.003375 (+ 2992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.003383 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.003502 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.003607 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:08.006809 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.008087 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.012422 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.013165 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.013300 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.018699 (+ 5399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:08.018707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:08.018714 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:08.018768 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:08.023011 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.024211 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.024218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.024222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.024354 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.025012 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.025020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.025026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.025447 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:08.025592 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.026304 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.026548 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:53:08.028342 10985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:07.999322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.8.204.130:56975)
0927 22:53:08.000189 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.000241 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.003022 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.004756 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.004784 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.005640 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:08.006608 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.006645 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.008285 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.008296 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.012212 (+ 3916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.012234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.013723 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.013733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.014107 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.014981 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:08.015002 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:08.018392 (+ 3390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:08.023174 (+ 4782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.023183 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.023203 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.023974 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.024507 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.024514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.024520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.024829 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.025574 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:08.025581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:08.025957 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.026206 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.026708 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":336,"threads_started":1}
I20260927 22:53:08.048977 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.033781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:58997 (local address 127.8.204.190:45711)
0927 22:53:08.033955 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.033963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.034557 (+ 594us) server_negotiation.cc:407] Connection header received
0927 22:53:08.034741 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.034744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.034824 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.034894 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:08.036539 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.037197 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.039613 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.040044 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.040134 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.042491 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:08.042498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:08.042503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:08.042543 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:08.046048 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.046664 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.046669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.046674 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.046823 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.047238 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.047246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.047252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.047689 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:08.047775 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.048442 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.048642 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:08.049116 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.033781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.8.204.130:58997)
0927 22:53:08.034443 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.034467 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.034625 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.034957 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.034964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.035754 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:08.036362 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.036380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.037350 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.037358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.039474 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.039492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.040240 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.040245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.040395 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.041023 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:08.041053 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:08.042276 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:08.046159 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.046162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.046164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.046551 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.046925 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.046928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.046930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.047091 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.047876 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:08.047883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:08.048136 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.048337 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.048660 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":322,"threads_started":1}
I20260927 22:53:08.055907 10961 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:41125
I20260927 22:53:08.055953 11095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:41125 every 8 connection(s)
I20260927 22:53:08.057096 10961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:08.058920 11096 heartbeater.cc:347] Connected to a master server at 127.8.204.190:45711
I20260927 22:53:08.059119 11096 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:08.059751 11096 heartbeater.cc:511] Master 127.8.204.190:45711 requested a full tablet report, sending...
I20260927 22:53:08.060878 10750 ts_manager.cc:194] Registered new tserver with Master: feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125)
I20260927 22:53:08.061618 10750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:56975
I20260927 22:53:08.064910 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 10961
I20260927 22:53:08.065073 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:08.065474 9010 external_mini_cluster.cc:1428] Setting key 7ef464a8aa8dc3ab9fe350c764344025
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:08.102284 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45711
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:40829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:08.411458 11100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:08.411720 11100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:08.411904 11100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:08.418983 11100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:08.419107 11100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:08.419260 11100 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:53:08.419368 11100 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:08.419504 11100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:08.429168 11100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:45711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:08.430541 11100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:08.431906 11100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:08.444347 11105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:08.444309 11106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:08.444269 11108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:08.445441 11100 server_base.cc:1034] running on GCE node
I20260927 22:53:08.445957 11100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:08.446854 11100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:08.448051 11100 hybrid_clock.cc:648] HybridClock initialized: now 1790549588448001 us; error 32 us; skew 500 ppm
I20260927 22:53:08.448369 11100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:08 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549588, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:08.451484 11100 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:08.452996 11100 webserver.cc:460] Webserver started at http://127.8.204.131:37219/ using document root <none> and password file <none>
I20260927 22:53:08.453719 11100 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:08.453856 11100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:08.454180 11100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:08.457103 11100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "a462ceabf15b466b9e13bbb5dd5fe347"
format_stamp: "Formatted at 2026-09-27 22:53:08 on dist-test-slave-40vx"
server_key: "474d6fb711796ad1fd1ca842b97fa3ce"
server_key_iv: "439906f430e302a80053b0942c03a9c5"
server_key_version: "encryptionkey@0"
I20260927 22:53:08.457852 11100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "a462ceabf15b466b9e13bbb5dd5fe347"
format_stamp: "Formatted at 2026-09-27 22:53:08 on dist-test-slave-40vx"
server_key: "474d6fb711796ad1fd1ca842b97fa3ce"
server_key_iv: "439906f430e302a80053b0942c03a9c5"
server_key_version: "encryptionkey@0"
I20260927 22:53:08.463042 11100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:53:08.466521 11115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:08.467991 11100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:08.468159 11100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "a462ceabf15b466b9e13bbb5dd5fe347"
format_stamp: "Formatted at 2026-09-27 22:53:08 on dist-test-slave-40vx"
server_key: "474d6fb711796ad1fd1ca842b97fa3ce"
server_key_iv: "439906f430e302a80053b0942c03a9c5"
server_key_version: "encryptionkey@0"
I20260927 22:53:08.468350 11100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:08.498337 11100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:08.502393 11100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:08.502678 11100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:08.505332 11100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:08.505482 11100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:08.505594 11100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:08.505674 11100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:08 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549588, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:08.531044 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.506065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:53847 (local address 127.8.204.190:45711)
0927 22:53:08.506256 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.506262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.507730 (+ 1468us) server_negotiation.cc:407] Connection header received
0927 22:53:08.509662 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.509667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.509747 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.509861 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:08.511993 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.513011 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.515680 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.516172 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.516301 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.520690 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:08.520700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:08.520707 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:08.520759 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:08.524467 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.525433 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.525440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.525446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.525606 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.526191 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.526196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.526199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.526586 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:08.526762 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.530283 (+ 3521us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.530573 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:53:08.531796 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.506710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.8.204.131:53847)
0927 22:53:08.507573 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.507607 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.509381 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.510086 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.510113 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.510963 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:08.511824 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.511846 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.513206 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.513212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.515470 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.515488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.516502 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.516515 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.516892 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.517708 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:08.517730 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:08.520434 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:08.524602 (+ 4168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.524610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.524625 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.525302 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.525721 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.525726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.525730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.526029 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.526743 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:08.526755 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:08.527350 (+ 595us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.527742 (+ 392us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.528431 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":391,"threads_started":1}
I20260927 22:53:08.555027 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.538186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:54471 (local address 127.8.204.190:45711)
0927 22:53:08.538380 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.538388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.539919 (+ 1531us) server_negotiation.cc:407] Connection header received
0927 22:53:08.540038 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.540045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.540139 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.540262 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:08.541079 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.542561 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.544227 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.544833 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.544978 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.547462 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:08.547469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:08.547475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:08.547516 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:08.551809 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.552634 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.552642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.552647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.552855 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.553250 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.553257 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.553262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.553711 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:08.553942 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.554297 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.554570 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:53:08.555105 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.537882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.8.204.131:54471)
0927 22:53:08.539827 (+ 1945us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.539842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.539945 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.540269 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.540271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.540687 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:08.540952 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.540960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.542704 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.542708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.544082 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.544092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.545044 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.545055 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.545241 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.545858 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:08.545883 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:08.547230 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:08.551957 (+ 4727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.551965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.551970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.552501 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.552976 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.552980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.552984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.553130 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.553891 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:08.553894 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:08.554063 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.554183 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.554509 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"thread_start_us":1606,"threads_started":1}
I20260927 22:53:08.559475 10957 heartbeater.cc:503] Master 127.8.204.190:45711 was elected leader, sending a full tablet report...
I20260927 22:53:08.561256 11100 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:39509
I20260927 22:53:08.561292 11234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:39509 every 8 connection(s)
I20260927 22:53:08.562388 11100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:08.564348 11235 heartbeater.cc:347] Connected to a master server at 127.8.204.190:45711
I20260927 22:53:08.564621 11235 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:08.565239 11235 heartbeater.cc:511] Master 127.8.204.190:45711 requested a full tablet report, sending...
I20260927 22:53:08.566367 10750 ts_manager.cc:194] Registered new tserver with Master: a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509)
I20260927 22:53:08.567178 10750 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:53847
I20260927 22:53:08.568766 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 11100
I20260927 22:53:08.568946 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:08.569468 9010 external_mini_cluster.cc:1428] Setting key 6d67459d3b5340fbd7368268935589e4
I20260927 22:53:08.572192 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:53:08 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:53:08 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:08.615145 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.601806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.8.204.190:45711)
0927 22:53:08.602025 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.602043 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.602243 (+ 200us) server_negotiation.cc:407] Connection header received
0927 22:53:08.602387 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.602390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.602454 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.602536 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:08.603896 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.604594 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.605977 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.606481 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.606578 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.610104 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:08.610108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:08.610111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:08.610141 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:08.612654 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.613277 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.613281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.613283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.613403 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.613774 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.613778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.613780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.614059 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:08.614256 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.614544 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.614702 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:53:08.618789 10750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:08.625558 10750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:08.652695 11250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.638136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.0.0.1:60054)
0927 22:53:08.640094 (+ 1958us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.640107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.640185 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.644268 (+ 4083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.644275 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.644306 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:08.644726 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.644743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.646851 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.646859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.649255 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.649273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.651165 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.651172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.651579 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.651639 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.651771 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.651949 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841,"thread_start_us":1721,"threads_started":1}
I20260927 22:53:08.652827 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.639640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.8.204.131:39509)
0927 22:53:08.643645 (+ 4005us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.643651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.643677 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:53:08.643757 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.643784 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.644046 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.644254 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:08.644999 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.646678 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.649437 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.650965 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.651095 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.651340 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.652331 (+ 991us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.652497 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3877,"thread_start_us":3725,"threads_started":1}
I20260927 22:53:08.653131 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.638742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:43495 (local address 127.0.0.1:47468)
0927 22:53:08.644999 (+ 6257us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.645016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.645129 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.645924 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.645929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.645959 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:08.646316 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.646325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.648625 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.648631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.650497 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.650507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.652159 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.652166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.652391 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.652452 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.652554 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.652757 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6120,"mutex_wait_us":2,"thread_start_us":5976,"threads_started":1}
I20260927 22:53:08.653880 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.640180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.8.204.129:43495)
0927 22:53:08.640651 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.640658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.645254 (+ 4596us) server_negotiation.cc:407] Connection header received
0927 22:53:08.645357 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.645364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.645585 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.645753 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:08.646487 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.648483 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.650632 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.651929 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.653154 (+ 1225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.653363 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.653439 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.653598 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260927 22:53:08.655296 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.637607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:41125 (local address 127.0.0.1:49070)
0927 22:53:08.639239 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.639311 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.639511 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.647221 (+ 7710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.647225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.647250 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:08.647636 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.647644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.649223 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.649229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.651489 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.651509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.654489 (+ 2980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.654496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.654663 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.654693 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.654781 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.654919 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":23,"thread_start_us":276,"threads_started":1}
I20260927 22:53:08.655308 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.637857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.8.204.130:41125)
0927 22:53:08.646805 (+ 8948us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.646815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.646841 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:53:08.646928 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.646933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.647061 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.647230 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:08.647800 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.649022 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.651656 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.654294 (+ 2638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.654441 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.654625 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.654848 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.655005 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8793,"thread_start_us":8678,"threads_started":1}
I20260927 22:53:08.656479 11165 tablet_service.cc:1467] Processing CreateTablet for tablet 0313fa920360486383900b15fddd5796 (DEFAULT_TABLE table=test-table [id=91d78465b0db49d9987e98bf82263be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:08.657646 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0313fa920360486383900b15fddd5796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:08.661190 10887 tablet_service.cc:1467] Processing CreateTablet for tablet 0313fa920360486383900b15fddd5796 (DEFAULT_TABLE table=test-table [id=91d78465b0db49d9987e98bf82263be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:08.662362 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0313fa920360486383900b15fddd5796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:08.665221 11026 tablet_service.cc:1467] Processing CreateTablet for tablet 0313fa920360486383900b15fddd5796 (DEFAULT_TABLE table=test-table [id=91d78465b0db49d9987e98bf82263be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:08.666456 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0313fa920360486383900b15fddd5796. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:08.668504 11254 tablet_bootstrap.cc:492] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Bootstrap starting.
I20260927 22:53:08.671415 11254 tablet_bootstrap.cc:654] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:08.672551 11254 log.cc:824] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:08.676169 11256 tablet_bootstrap.cc:492] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Bootstrap starting.
I20260927 22:53:08.679322 11254 tablet_bootstrap.cc:492] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: No bootstrap required, opened a new log
I20260927 22:53:08.679580 11254 ts_tablet_manager.cc:1397] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260927 22:53:08.680009 11256 tablet_bootstrap.cc:654] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:08.679687 11257 tablet_bootstrap.cc:492] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Bootstrap starting.
I20260927 22:53:08.681479 11256 log.cc:824] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:08.683054 11257 tablet_bootstrap.cc:654] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:08.683966 11257 log.cc:824] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:08.686108 11257 tablet_bootstrap.cc:492] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: No bootstrap required, opened a new log
I20260927 22:53:08.686374 11257 ts_tablet_manager.cc:1397] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 22:53:08.688551 11256 tablet_bootstrap.cc:492] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: No bootstrap required, opened a new log
I20260927 22:53:08.688845 11256 ts_tablet_manager.cc:1397] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20260927 22:53:08.690934 11254 raft_consensus.cc:348] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.691390 11254 raft_consensus.cc:374] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:08.691485 11254 raft_consensus.cc:729] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Initialized, Role: FOLLOWER
I20260927 22:53:08.692072 11254 consensus_queue.cc:260] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.693140 11235 heartbeater.cc:503] Master 127.8.204.190:45711 was elected leader, sending a full tablet report...
I20260927 22:53:08.693747 11254 ts_tablet_manager.cc:1428] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260927 22:53:08.695813 11257 raft_consensus.cc:348] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.696233 11257 raft_consensus.cc:374] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:08.696347 11257 raft_consensus.cc:729] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7399116844192a4a38bf6d102cbf6, State: Initialized, Role: FOLLOWER
I20260927 22:53:08.696889 11257 consensus_queue.cc:260] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.697871 11096 heartbeater.cc:503] Master 127.8.204.190:45711 was elected leader, sending a full tablet report...
I20260927 22:53:08.697795 11256 raft_consensus.cc:348] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.698160 11256 raft_consensus.cc:374] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:08.698244 11256 raft_consensus.cc:729] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9987f4c9e0403d824db44b0c859525, State: Initialized, Role: FOLLOWER
I20260927 22:53:08.698475 11257 ts_tablet_manager.cc:1428] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 22:53:08.698928 11256 consensus_queue.cc:260] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.700515 11256 ts_tablet_manager.cc:1428] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:53:08.749320 11260 raft_consensus.cc:482] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:08.749650 11260 raft_consensus.cc:504] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.751210 11260 leader_election.cc:288] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125), 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495)
I20260927 22:53:08.757776 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.751632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:41125 (local address 127.8.204.131:36407)
0927 22:53:08.751849 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.751866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.752027 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.752509 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.752514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.752547 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:08.752736 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.752744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.754041 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.754047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.756009 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.756023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.757230 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.757234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.757353 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.757377 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.757485 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.757573 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 22:53:08.757951 11249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.751632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:36407 (local address 127.8.204.130:41125)
0927 22:53:08.752195 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.752199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.752234 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:53:08.752303 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.752308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.752382 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.752476 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:08.752870 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.753923 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.756139 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.757105 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.757261 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.757336 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.757525 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.757673 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410}
I20260927 22:53:08.758580 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.751934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:43495 (local address 127.8.204.131:33625)
0927 22:53:08.752710 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.752723 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.752860 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.753211 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.753215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.753242 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:08.753572 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.753582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.754616 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.754621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.756461 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.756470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.757965 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.757970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.758140 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.758171 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.758266 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.758357 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":488,"threads_started":1}
I20260927 22:53:08.758661 11253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.752163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:33625 (local address 127.8.204.129:43495)
0927 22:53:08.752371 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.752377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.752802 (+ 425us) server_negotiation.cc:407] Connection header received
0927 22:53:08.752988 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.752992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.753056 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.753174 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:08.753719 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.754488 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.756601 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.757752 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.757912 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.757989 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.758301 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.758441 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:53:08.759143 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0313fa920360486383900b15fddd5796" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6" is_pre_election: true
I20260927 22:53:08.759660 11046 raft_consensus.cc:2454] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 0.
I20260927 22:53:08.759639 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0313fa920360486383900b15fddd5796" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525" is_pre_election: true
I20260927 22:53:08.760128 10907 raft_consensus.cc:2454] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 0.
I20260927 22:53:08.760308 11119 leader_election.cc:302] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a462ceabf15b466b9e13bbb5dd5fe347, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:08.760712 11260 raft_consensus.cc:2781] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:08.760856 11260 raft_consensus.cc:482] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:08.760972 11260 raft_consensus.cc:3037] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:08.763048 11260 raft_consensus.cc:504] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.763693 11260 leader_election.cc:288] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 election: Requested vote from peers feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125), 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495)
I20260927 22:53:08.764027 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0313fa920360486383900b15fddd5796" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6"
I20260927 22:53:08.764074 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0313fa920360486383900b15fddd5796" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525"
I20260927 22:53:08.764222 11046 raft_consensus.cc:3037] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:08.764238 10907 raft_consensus.cc:3037] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:08.767316 11046 raft_consensus.cc:2454] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 1.
I20260927 22:53:08.767316 10907 raft_consensus.cc:2454] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 1.
I20260927 22:53:08.767658 11119 leader_election.cc:302] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e9987f4c9e0403d824db44b0c859525, a462ceabf15b466b9e13bbb5dd5fe347; no voters:
I20260927 22:53:08.767927 11260 raft_consensus.cc:2781] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:08.768323 11260 raft_consensus.cc:686] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 LEADER]: Becoming Leader. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Running, Role: LEADER
I20260927 22:53:08.768764 11260 consensus_queue.cc:237] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.772925 10749 catalog_manager.cc:5495] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 reported cstate change: term changed from 0 to 1, leader changed from <none> to a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } health_report { overall_health: HEALTHY } } }
W20260927 22:53:08.802834 10958 tablet.cc:2369] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:08.808971 11097 tablet.cc:2369] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:08.813876 11236 tablet.cc:2369] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:53:08 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549586, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:08.837966 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.825594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45711 (local address 127.0.0.1:42410)
0927 22:53:08.826195 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.826210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.826376 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.826673 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.826676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.827424 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:08.827694 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.827702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.828606 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.828608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.829558 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.829568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.830453 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.830457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.830572 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.831206 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:08.831226 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:08.833337 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:08.835962 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.835972 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.835977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.836356 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.836650 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:08.836652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:08.836655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:08.836779 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:08.837197 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:08.837203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:08.837338 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.837461 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.837650 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"spinlock_wait_cycles":16256,"thread_start_us":328,"threads_started":1}
I20260927 22:53:08.837994 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.825566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.8.204.190:45711)
0927 22:53:08.825748 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.825757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.826317 (+ 560us) server_negotiation.cc:407] Connection header received
0927 22:53:08.826481 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.826484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.826556 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.826636 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:08.827815 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.828508 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.829721 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.830286 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.830444 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.833535 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:08.833540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:08.833545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:08.833572 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:08.835845 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.836447 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.836451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.836453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.836557 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:08.836859 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:08.836862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:08.836864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:08.837105 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:08.837203 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.837535 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.837757 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":59}
I20260927 22:53:08.842558 10749 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:42410:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:53:08.843631 10749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:08.849797 10887 tablet_service.cc:1467] Processing CreateTablet for tablet 55ca03bc2e144eb8905277b7dea46df9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f294aca105804e9ba193c4a019a0199e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:08.850092 11165 tablet_service.cc:1467] Processing CreateTablet for tablet 55ca03bc2e144eb8905277b7dea46df9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f294aca105804e9ba193c4a019a0199e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:08.850095 11026 tablet_service.cc:1467] Processing CreateTablet for tablet 55ca03bc2e144eb8905277b7dea46df9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f294aca105804e9ba193c4a019a0199e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:08.850642 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ca03bc2e144eb8905277b7dea46df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:08.850924 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ca03bc2e144eb8905277b7dea46df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:08.850921 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ca03bc2e144eb8905277b7dea46df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:08.856204 11256 tablet_bootstrap.cc:492] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: Bootstrap starting.
I20260927 22:53:08.857497 11256 tablet_bootstrap.cc:654] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:08.857836 11254 tablet_bootstrap.cc:492] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: Bootstrap starting.
I20260927 22:53:08.859130 11257 tablet_bootstrap.cc:492] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: Bootstrap starting.
I20260927 22:53:08.859542 11254 tablet_bootstrap.cc:654] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:08.859742 11256 tablet_bootstrap.cc:492] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: No bootstrap required, opened a new log
I20260927 22:53:08.859886 11256 ts_tablet_manager.cc:1397] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:53:08.860698 11257 tablet_bootstrap.cc:654] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:08.860589 11256 raft_consensus.cc:348] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.860817 11256 raft_consensus.cc:374] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:08.860913 11256 raft_consensus.cc:729] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9987f4c9e0403d824db44b0c859525, State: Initialized, Role: FOLLOWER
I20260927 22:53:08.861143 11256 consensus_queue.cc:260] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.861630 11262 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.861899 11256 ts_tablet_manager.cc:1428] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 22:53:08.862007 11262 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:08.862457 11254 tablet_bootstrap.cc:492] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: No bootstrap required, opened a new log
I20260927 22:53:08.862596 11254 ts_tablet_manager.cc:1397] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 22:53:08.863189 11254 raft_consensus.cc:348] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.863453 11254 raft_consensus.cc:374] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:08.863559 11254 raft_consensus.cc:729] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Initialized, Role: FOLLOWER
I20260927 22:53:08.863791 11254 consensus_queue.cc:260] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.864221 11260 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.864408 11254 ts_tablet_manager.cc:1428] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:08.864517 11260 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:08.866151 11257 tablet_bootstrap.cc:492] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: No bootstrap required, opened a new log
I20260927 22:53:08.866289 11257 ts_tablet_manager.cc:1397] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260927 22:53:08.866854 11257 raft_consensus.cc:348] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.867100 11257 raft_consensus.cc:374] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:08.867198 11257 raft_consensus.cc:729] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7399116844192a4a38bf6d102cbf6, State: Initialized, Role: FOLLOWER
I20260927 22:53:08.867453 11257 consensus_queue.cc:260] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.868086 11257 ts_tablet_manager.cc:1428] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:53:08.867942 11261 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.868356 11261 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:08.922147 11260 raft_consensus.cc:482] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:08.922317 11260 raft_consensus.cc:504] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.922900 11260 leader_election.cc:288] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125)
I20260927 22:53:08.923547 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca03bc2e144eb8905277b7dea46df9" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525" is_pre_election: true
I20260927 22:53:08.923540 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca03bc2e144eb8905277b7dea46df9" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6" is_pre_election: true
I20260927 22:53:08.923746 10907 raft_consensus.cc:2454] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 0.
I20260927 22:53:08.923782 11046 raft_consensus.cc:2454] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 0.
I20260927 22:53:08.924187 11119 leader_election.cc:302] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a462ceabf15b466b9e13bbb5dd5fe347, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:08.924547 11260 raft_consensus.cc:2781] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:08.924651 11260 raft_consensus.cc:482] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:08.924743 11260 raft_consensus.cc:3037] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:08.926801 11260 raft_consensus.cc:504] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.927351 11260 leader_election.cc:288] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 election: Requested vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125)
I20260927 22:53:08.927835 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca03bc2e144eb8905277b7dea46df9" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525"
I20260927 22:53:08.927835 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca03bc2e144eb8905277b7dea46df9" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6"
I20260927 22:53:08.928036 11046 raft_consensus.cc:3037] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:08.928037 10907 raft_consensus.cc:3037] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:08.931012 10907 raft_consensus.cc:2454] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 1.
I20260927 22:53:08.931012 11046 raft_consensus.cc:2454] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 1.
I20260927 22:53:08.931432 11119 leader_election.cc:302] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a462ceabf15b466b9e13bbb5dd5fe347, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:08.931767 11260 raft_consensus.cc:2781] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:08.931973 11260 raft_consensus.cc:686] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 LEADER]: Becoming Leader. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Running, Role: LEADER
I20260927 22:53:08.932224 11260 consensus_queue.cc:237] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:08.933954 11264 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: TxnStatusTablet state changed. Reason: New leader a462ceabf15b466b9e13bbb5dd5fe347. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.934366 11264 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:08.935086 11276 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:53:08.935413 11276 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:53:08.935528 10750 catalog_manager.cc:5495] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 reported cstate change: term changed from 0 to 1, leader changed from <none> to a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } health_report { overall_health: HEALTHY } } }
I20260927 22:53:08.965271 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.959191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.0.0.1:60058)
0927 22:53:08.959432 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:08.959504 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:08.959638 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:08.960132 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:08.960135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:08.960144 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:08.960303 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.960309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.961652 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.961655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.962612 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:08.962622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.963604 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.963609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:08.963792 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.963860 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:08.964868 (+ 1008us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:08.964883 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 22:53:08.964939 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 22:53:08.965047 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:53:08.965338 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:08.959330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.8.204.131:39509)
0927 22:53:08.959542 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:08.959548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:08.959570 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:53:08.959829 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:08.959835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:08.959953 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:08.960133 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:08.960428 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.961535 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.962738 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:08.963454 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:08.963588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:08.964052 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:08.964757 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:08.964907 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:08.964965 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:53:08.965047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:53:08.973248 11046 raft_consensus.cc:1261] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Refusing update from remote peer a462ceabf15b466b9e13bbb5dd5fe347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:08.973248 10907 raft_consensus.cc:1261] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Refusing update from remote peer a462ceabf15b466b9e13bbb5dd5fe347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:08.974208 11264 consensus_queue.cc:1035] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:08.974586 11260 consensus_queue.cc:1035] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:08.983327 11261 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: TxnStatusTablet state changed. Reason: New leader a462ceabf15b466b9e13bbb5dd5fe347. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.983595 11261 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:08.986558 11279 mvcc.cc:205] Tried to move back new op lower bound from 7334091116423618560 to 7334091116268515328. Current Snapshot: MvccSnapshot[applied={T|T < 7334091116423618560}]
I20260927 22:53:08.989621 11260 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.990024 11260 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:08.997449 11261 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:08.997692 11261 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P feb7399116844192a4a38bf6d102cbf6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:09.000584 11262 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: TxnStatusTablet state changed. Reason: New leader a462ceabf15b466b9e13bbb5dd5fe347. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:09.000857 11262 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:09.003480 11264 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:09.003836 11264 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P a462ceabf15b466b9e13bbb5dd5fe347: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:09.007985 11262 tablet_replica.cc:439] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } } }
I20260927 22:53:09.008234 11262 tablet_replica.cc:442] T 55ca03bc2e144eb8905277b7dea46df9 P 9e9987f4c9e0403d824db44b0c859525: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:09.014278 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.007502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60062 (local address 127.8.204.131:39509)
0927 22:53:09.008500 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:09.008507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:09.009853 (+ 1346us) server_negotiation.cc:407] Connection header received
0927 22:53:09.009913 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:09.009917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:09.009977 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:09.010073 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:09.010428 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.011193 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.012634 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.013045 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.013222 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.013618 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:09.013720 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:09.013815 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:09.013933 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 22:53:09.014007 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875}
I20260927 22:53:09.029407 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.024090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.8.204.131:42931)
0927 22:53:09.024293 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:09.024313 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:09.024472 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:09.024748 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:09.024753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:09.024762 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:09.024998 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:09.025006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.025992 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.025999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:09.027195 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:09.027206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.028203 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.028207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:09.028351 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.028466 (+ 115us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:09.028897 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:09.028905 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:53:09.028963 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 22:53:09.029124 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":99}
I20260927 22:53:09.029414 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.024170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:42931 (local address 127.8.204.131:39509)
0927 22:53:09.024293 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:09.024299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:09.024383 (+ 84us) server_negotiation.cc:407] Connection header received
0927 22:53:09.024588 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:09.024592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:09.024651 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:09.024730 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:09.025105 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.025871 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.027378 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.028050 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.028193 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.028650 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:09.028798 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:09.028879 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:09.029003 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 22:53:09.029112 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":49}
I20260927 22:53:09.038655 10907 raft_consensus.cc:1261] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Refusing update from remote peer a462ceabf15b466b9e13bbb5dd5fe347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:09.038762 11046 raft_consensus.cc:1261] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Refusing update from remote peer a462ceabf15b466b9e13bbb5dd5fe347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:09.039160 11287 consensus_queue.cc:1035] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:09.039505 11260 consensus_queue.cc:1035] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:09.066919 11281 mvcc.cc:205] Tried to move back new op lower bound from 7334091116697907200 to 7334091115600191488. Current Snapshot: MvccSnapshot[applied={T|T < 7334091116697907200}]
I20260927 22:53:09.110954 10750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:09.111230 10750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:09.114354 10750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Sending DeleteTablet for 3 replicas of tablet 0313fa920360486383900b15fddd5796
I20260927 22:53:09.115447 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet 0313fa920360486383900b15fddd5796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:09 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49070
I20260927 22:53:09.115535 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet 0313fa920360486383900b15fddd5796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:09 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:47468
I20260927 22:53:09.116343 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet 0313fa920360486383900b15fddd5796 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:09 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:60054
I20260927 22:53:09.116721 10750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:09.117641 10750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:09.120018 11297 tablet_replica.cc:330] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: stopping tablet replica
I20260927 22:53:09.120472 11297 raft_consensus.cc:2229] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:09.123154 11297 raft_consensus.cc:2258] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:09.124018 11298 tablet_replica.cc:330] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: stopping tablet replica
I20260927 22:53:09.124953 11165 tablet_service.cc:1467] Processing CreateTablet for tablet 380dfa4da54c4c43af0168e325880d56 (DEFAULT_TABLE table=test-table [id=5ec47abda3064a119792bf704b87f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:09.125442 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380dfa4da54c4c43af0168e325880d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:09.125443 11025 tablet_service.cc:1467] Processing CreateTablet for tablet 380dfa4da54c4c43af0168e325880d56 (DEFAULT_TABLE table=test-table [id=5ec47abda3064a119792bf704b87f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:09.125728 10887 tablet_service.cc:1467] Processing CreateTablet for tablet 380dfa4da54c4c43af0168e325880d56 (DEFAULT_TABLE table=test-table [id=5ec47abda3064a119792bf704b87f741]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:09.125968 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380dfa4da54c4c43af0168e325880d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:09.126237 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380dfa4da54c4c43af0168e325880d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:09.132524 11256 tablet_bootstrap.cc:492] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Bootstrap starting.
I20260927 22:53:09.134264 11256 tablet_bootstrap.cc:654] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:09.134429 11257 tablet_bootstrap.cc:492] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Bootstrap starting.
I20260927 22:53:09.134747 11297 ts_tablet_manager.cc:1905] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:09.135877 11298 raft_consensus.cc:2229] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:09.136260 11257 tablet_bootstrap.cc:654] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:09.136487 11298 raft_consensus.cc:2258] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:09.139818 11296 tablet_replica.cc:330] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: stopping tablet replica
I20260927 22:53:09.140216 11296 raft_consensus.cc:2229] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:09.140570 11254 tablet_bootstrap.cc:492] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Bootstrap starting.
I20260927 22:53:09.140560 11296 raft_consensus.cc:2258] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:09.140837 11298 ts_tablet_manager.cc:1905] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:09.142228 11254 tablet_bootstrap.cc:654] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:09.149335 11257 tablet_bootstrap.cc:492] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: No bootstrap required, opened a new log
I20260927 22:53:09.149526 11257 ts_tablet_manager.cc:1397] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20260927 22:53:09.150103 11257 raft_consensus.cc:348] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.150309 11257 raft_consensus.cc:374] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:09.150379 11257 raft_consensus.cc:729] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7399116844192a4a38bf6d102cbf6, State: Initialized, Role: FOLLOWER
I20260927 22:53:09.150628 11257 consensus_queue.cc:260] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.151257 11257 ts_tablet_manager.cc:1428] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:53:09.151643 11297 ts_tablet_manager.cc:1918] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:09.151799 11297 log.cc:1196] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/0313fa920360486383900b15fddd5796
I20260927 22:53:09.152304 11297 ts_tablet_manager.cc:1939] T 0313fa920360486383900b15fddd5796 P 9e9987f4c9e0403d824db44b0c859525: Deleting consensus metadata
I20260927 22:53:09.153208 11296 ts_tablet_manager.cc:1905] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:09.154023 10737 catalog_manager.cc:4841] TS 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495): tablet 0313fa920360486383900b15fddd5796 (table test-table [id=91d78465b0db49d9987e98bf82263be7]) successfully deleted
I20260927 22:53:09.154444 11256 tablet_bootstrap.cc:492] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: No bootstrap required, opened a new log
I20260927 22:53:09.154593 11256 ts_tablet_manager.cc:1397] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.005s
I20260927 22:53:09.155349 11256 raft_consensus.cc:348] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.155573 11256 raft_consensus.cc:374] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:09.155652 11256 raft_consensus.cc:729] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9987f4c9e0403d824db44b0c859525, State: Initialized, Role: FOLLOWER
I20260927 22:53:09.155896 11256 consensus_queue.cc:260] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.156461 11256 ts_tablet_manager.cc:1428] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:09.157956 11254 tablet_bootstrap.cc:492] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: No bootstrap required, opened a new log
I20260927 22:53:09.158147 11254 ts_tablet_manager.cc:1397] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20260927 22:53:09.158911 11254 raft_consensus.cc:348] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.159159 11254 raft_consensus.cc:374] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:09.159232 11254 raft_consensus.cc:729] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Initialized, Role: FOLLOWER
I20260927 22:53:09.159472 11298 ts_tablet_manager.cc:1918] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:09.159651 11298 log.cc:1196] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/0313fa920360486383900b15fddd5796
I20260927 22:53:09.159655 11254 consensus_queue.cc:260] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.160362 11254 ts_tablet_manager.cc:1428] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:09.162436 11298 ts_tablet_manager.cc:1939] T 0313fa920360486383900b15fddd5796 P a462ceabf15b466b9e13bbb5dd5fe347: Deleting consensus metadata
I20260927 22:53:09.164127 10734 catalog_manager.cc:4841] TS a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509): tablet 0313fa920360486383900b15fddd5796 (table test-table [id=91d78465b0db49d9987e98bf82263be7]) successfully deleted
I20260927 22:53:09.165999 11296 ts_tablet_manager.cc:1918] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:09.166157 11296 log.cc:1196] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/0313fa920360486383900b15fddd5796
I20260927 22:53:09.166584 11296 ts_tablet_manager.cc:1939] T 0313fa920360486383900b15fddd5796 P feb7399116844192a4a38bf6d102cbf6: Deleting consensus metadata
I20260927 22:53:09.167982 10737 catalog_manager.cc:4841] TS feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125): tablet 0313fa920360486383900b15fddd5796 (table test-table [id=91d78465b0db49d9987e98bf82263be7]) successfully deleted
I20260927 22:53:09.318588 11261 raft_consensus.cc:482] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:09.318934 11261 raft_consensus.cc:504] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.320430 11261 leader_election.cc:288] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509)
I20260927 22:53:09.331205 11260 raft_consensus.cc:482] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:09.331432 11260 raft_consensus.cc:504] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.332217 11260 leader_election.cc:288] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125)
I20260927 22:53:09.333835 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380dfa4da54c4c43af0168e325880d56" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6" is_pre_election: true
I20260927 22:53:09.333352 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380dfa4da54c4c43af0168e325880d56" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525" is_pre_election: true
I20260927 22:53:09.334105 11045 raft_consensus.cc:2454] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 0.
I20260927 22:53:09.334192 10907 raft_consensus.cc:2454] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 0.
I20260927 22:53:09.334918 11119 leader_election.cc:302] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a462ceabf15b466b9e13bbb5dd5fe347, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:09.335465 11260 raft_consensus.cc:2781] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:09.335610 11260 raft_consensus.cc:482] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:09.335711 11260 raft_consensus.cc:3037] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:09.339323 11260 raft_consensus.cc:504] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.340312 10907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380dfa4da54c4c43af0168e325880d56" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525"
I20260927 22:53:09.340530 10907 raft_consensus.cc:3037] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:09.343243 11304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.320786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:43495 (local address 127.8.204.130:55771)
0927 22:53:09.333318 (+ 12532us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:09.333348 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:09.333525 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:09.335547 (+ 2022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:09.335555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:09.335589 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:09.336031 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:09.336043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.337587 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.337596 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:09.340206 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:09.340222 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.342277 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.342286 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:09.342508 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.342546 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:53:09.342632 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:53:09.342791 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12351,"mutex_wait_us":77,"thread_start_us":13038,"threads_started":1}
I20260927 22:53:09.343250 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.321216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:43607 (local address 127.8.204.131:39509)
0927 22:53:09.321403 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:09.321409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:09.334323 (+ 12914us) server_negotiation.cc:407] Connection header received
0927 22:53:09.334429 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:09.334434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:09.334512 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:09.334627 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:09.335776 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.337221 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.340396 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.341940 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.342077 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.342178 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:09.342791 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:53:09.342954 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:53:09.343621 10907 raft_consensus.cc:2454] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 1.
I20260927 22:53:09.344058 11302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.320808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:55771 (local address 127.8.204.129:43495)
0927 22:53:09.334936 (+ 14128us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:09.334943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:09.334960 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:53:09.335027 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:09.335032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:09.335114 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:09.335235 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:09.336241 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.337435 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.340421 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.341899 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.342093 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.342178 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:09.343515 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0927 22:53:09.343678 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13811,"spinlock_wait_cycles":1659264,"thread_start_us":13838,"threads_started":1}
I20260927 22:53:09.344060 11119 leader_election.cc:302] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e9987f4c9e0403d824db44b0c859525, a462ceabf15b466b9e13bbb5dd5fe347; no voters:
I20260927 22:53:09.344681 11303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:09.320951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.8.204.130:43607)
0927 22:53:09.334069 (+ 13118us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:09.334094 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:09.334242 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:09.334945 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:09.334951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:09.334997 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:09.335422 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:09.335438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.337483 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.337492 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:09.340215 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:09.340224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:09.342436 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:09.342443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:09.342613 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:09.342647 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:53:09.343902 (+ 1255us) client_negotiation.cc:240] Negotiation successful
0927 22:53:09.344051 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12970,"mutex_wait_us":78,"thread_start_us":12687,"threads_started":1}
I20260927 22:53:09.344748 11287 raft_consensus.cc:2781] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:09.344775 10906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380dfa4da54c4c43af0168e325880d56" candidate_uuid: "feb7399116844192a4a38bf6d102cbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525" is_pre_election: true
I20260927 22:53:09.344913 11287 raft_consensus.cc:686] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 LEADER]: Becoming Leader. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Running, Role: LEADER
I20260927 22:53:09.345127 10906 raft_consensus.cc:2379] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb7399116844192a4a38bf6d102cbf6 in current term 1: Already voted for candidate a462ceabf15b466b9e13bbb5dd5fe347 in this term.
I20260927 22:53:09.345165 11287 consensus_queue.cc:237] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:09.346604 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380dfa4da54c4c43af0168e325880d56" candidate_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6"
I20260927 22:53:09.346665 11260 leader_election.cc:288] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [CANDIDATE]: Term 1 election: Requested vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125)
I20260927 22:53:09.346808 11045 raft_consensus.cc:3037] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:09.346701 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380dfa4da54c4c43af0168e325880d56" candidate_uuid: "feb7399116844192a4a38bf6d102cbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" is_pre_election: true
I20260927 22:53:09.348729 10977 leader_election.cc:302] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb7399116844192a4a38bf6d102cbf6; no voters: 9e9987f4c9e0403d824db44b0c859525, a462ceabf15b466b9e13bbb5dd5fe347
I20260927 22:53:09.349915 11045 raft_consensus.cc:2454] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a462ceabf15b466b9e13bbb5dd5fe347 in term 1.
I20260927 22:53:09.350073 11261 raft_consensus.cc:2726] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:53:09.350081 10748 catalog_manager.cc:5495] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 reported cstate change: term changed from 0 to 1, leader changed from <none> to a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } health_report { overall_health: HEALTHY } } }
I20260927 22:53:09.390945 10907 raft_consensus.cc:1261] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Refusing update from remote peer a462ceabf15b466b9e13bbb5dd5fe347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:09.391009 11045 raft_consensus.cc:1261] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Refusing update from remote peer a462ceabf15b466b9e13bbb5dd5fe347: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:09.391708 11260 consensus_queue.cc:1035] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:09.391956 11293 consensus_queue.cc:1035] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:09.397092 11280 mvcc.cc:205] Tried to move back new op lower bound from 7334091118140039168 to 7334091117960040448. Current Snapshot: MvccSnapshot[applied={T|T < 7334091118140039168}]
W20260927 22:53:09.412567 11279 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:53:11.003129 10748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:11.003434 10748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:11.007831 10748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Sending DeleteTablet for 3 replicas of tablet 380dfa4da54c4c43af0168e325880d56
I20260927 22:53:11.008634 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet 380dfa4da54c4c43af0168e325880d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:11 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:47468
I20260927 22:53:11.008656 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet 380dfa4da54c4c43af0168e325880d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:11 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:60054
I20260927 22:53:11.008720 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet 380dfa4da54c4c43af0168e325880d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:11 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49070
I20260927 22:53:11.021034 11330 tablet_replica.cc:330] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: stopping tablet replica
I20260927 22:53:11.021027 11328 tablet_replica.cc:330] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: stopping tablet replica
I20260927 22:53:11.021245 11328 raft_consensus.cc:2229] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:11.021513 11328 raft_consensus.cc:2258] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:11.021673 11330 raft_consensus.cc:2229] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:11.021975 11330 raft_consensus.cc:2258] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:11.022127 11329 tablet_replica.cc:330] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: stopping tablet replica
I20260927 22:53:11.022312 11329 raft_consensus.cc:2229] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:11.022710 11329 raft_consensus.cc:2258] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:11.023130 11330 ts_tablet_manager.cc:1905] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:11.023180 11328 ts_tablet_manager.cc:1905] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:11.023559 11329 ts_tablet_manager.cc:1905] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:11.030447 11329 ts_tablet_manager.cc:1918] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:53:11.030596 11329 log.cc:1196] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/380dfa4da54c4c43af0168e325880d56
I20260927 22:53:11.030573 11330 ts_tablet_manager.cc:1918] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:53:11.030699 11330 log.cc:1196] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/380dfa4da54c4c43af0168e325880d56
I20260927 22:53:11.030989 11328 ts_tablet_manager.cc:1918] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:53:11.031052 11329 ts_tablet_manager.cc:1939] T 380dfa4da54c4c43af0168e325880d56 P a462ceabf15b466b9e13bbb5dd5fe347: Deleting consensus metadata
I20260927 22:53:11.031140 11330 ts_tablet_manager.cc:1939] T 380dfa4da54c4c43af0168e325880d56 P feb7399116844192a4a38bf6d102cbf6: Deleting consensus metadata
I20260927 22:53:11.031323 11328 log.cc:1196] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/380dfa4da54c4c43af0168e325880d56
I20260927 22:53:11.031800 11328 ts_tablet_manager.cc:1939] T 380dfa4da54c4c43af0168e325880d56 P 9e9987f4c9e0403d824db44b0c859525: Deleting consensus metadata
I20260927 22:53:11.032152 10734 catalog_manager.cc:4841] TS a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509): tablet 380dfa4da54c4c43af0168e325880d56 (table test-table [id=5ec47abda3064a119792bf704b87f741]) successfully deleted
I20260927 22:53:11.032156 10737 catalog_manager.cc:4841] TS feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125): tablet 380dfa4da54c4c43af0168e325880d56 (table test-table [id=5ec47abda3064a119792bf704b87f741]) successfully deleted
I20260927 22:53:11.032970 10737 catalog_manager.cc:4841] TS 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495): tablet 380dfa4da54c4c43af0168e325880d56 (table test-table [id=5ec47abda3064a119792bf704b87f741]) successfully deleted
Sep 27 22:53:11 dist-test-slave-40vx krb5kdc[10705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:53:11 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:11.065083 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.051150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.8.204.190:45711)
0927 22:53:11.051724 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.051728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.051834 (+ 106us) server_negotiation.cc:407] Connection header received
0927 22:53:11.051977 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.051981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.052045 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.052131 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:11.053401 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.054208 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.055460 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.055945 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.056084 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.060170 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:11.060175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:11.060179 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:11.060209 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:11.062688 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:11.063467 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:11.063471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:11.063473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:11.063593 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:11.063942 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:11.063944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:11.063946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:11.064200 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:11.064368 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.064639 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.064796 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":320,"threads_started":1}
I20260927 22:53:11.068538 10748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:11.069221 10748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:11.076115 10887 tablet_service.cc:1467] Processing CreateTablet for tablet cc2700c7b15b45389962cdb8b021028d (DEFAULT_TABLE table=test-table [id=0da170f2603e4610a86d6223617d57f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:11.076257 11026 tablet_service.cc:1467] Processing CreateTablet for tablet cc2700c7b15b45389962cdb8b021028d (DEFAULT_TABLE table=test-table [id=0da170f2603e4610a86d6223617d57f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:11.076558 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2700c7b15b45389962cdb8b021028d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:11.076511 11165 tablet_service.cc:1467] Processing CreateTablet for tablet cc2700c7b15b45389962cdb8b021028d (DEFAULT_TABLE table=test-table [id=0da170f2603e4610a86d6223617d57f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:11.076658 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2700c7b15b45389962cdb8b021028d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:11.076925 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc2700c7b15b45389962cdb8b021028d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:11.082278 11339 tablet_bootstrap.cc:492] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Bootstrap starting.
I20260927 22:53:11.083305 11340 tablet_bootstrap.cc:492] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Bootstrap starting.
I20260927 22:53:11.083954 11339 tablet_bootstrap.cc:654] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:11.085448 11340 tablet_bootstrap.cc:654] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:11.086426 11341 tablet_bootstrap.cc:492] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Bootstrap starting.
I20260927 22:53:11.087383 11339 tablet_bootstrap.cc:492] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: No bootstrap required, opened a new log
I20260927 22:53:11.087568 11339 ts_tablet_manager.cc:1397] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:53:11.088563 11341 tablet_bootstrap.cc:654] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:11.088534 11339 raft_consensus.cc:348] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.088830 11339 raft_consensus.cc:374] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:11.088939 11339 raft_consensus.cc:729] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9987f4c9e0403d824db44b0c859525, State: Initialized, Role: FOLLOWER
I20260927 22:53:11.089197 11339 consensus_queue.cc:260] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.093824 11340 tablet_bootstrap.cc:492] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: No bootstrap required, opened a new log
I20260927 22:53:11.094010 11340 ts_tablet_manager.cc:1397] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260927 22:53:11.094992 11340 raft_consensus.cc:348] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.095322 11340 raft_consensus.cc:374] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:11.095427 11340 raft_consensus.cc:729] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7399116844192a4a38bf6d102cbf6, State: Initialized, Role: FOLLOWER
I20260927 22:53:11.095746 11340 consensus_queue.cc:260] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.097152 11339 ts_tablet_manager.cc:1428] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260927 22:53:11.097314 11341 tablet_bootstrap.cc:492] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: No bootstrap required, opened a new log
I20260927 22:53:11.097462 11341 ts_tablet_manager.cc:1397] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260927 22:53:11.097980 11340 ts_tablet_manager.cc:1428] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260927 22:53:11.098286 11341 raft_consensus.cc:348] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.098539 11341 raft_consensus.cc:374] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:11.098636 11341 raft_consensus.cc:729] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Initialized, Role: FOLLOWER
I20260927 22:53:11.098872 11341 consensus_queue.cc:260] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.099416 11341 ts_tablet_manager.cc:1428] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:53:11.200775 11345 raft_consensus.cc:482] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:11.201020 11345 raft_consensus.cc:504] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.202569 11345 leader_election.cc:288] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125), a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509)
I20260927 22:53:11.223703 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.203100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:48573 (local address 127.8.204.131:39509)
0927 22:53:11.218384 (+ 15284us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.218388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.218409 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:11.218559 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.218562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.218620 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.218730 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:11.219368 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.220151 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.221925 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.222697 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.222883 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.222959 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.223198 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.223345 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15164,"thread_start_us":15292,"threads_started":1}
I20260927 22:53:11.223763 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.202859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.8.204.129:48573)
0927 22:53:11.218266 (+ 15407us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:11.218286 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:11.218412 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:11.218750 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:11.218754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:11.218810 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:11.219202 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.219211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.220271 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.220278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.221779 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.221790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.222860 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.222866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.223043 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.223069 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:53:11.223175 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:53:11.223321 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15260,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":15124,"threads_started":1}
I20260927 22:53:11.223960 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.202864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:43489 (local address 127.8.204.130:41125)
0927 22:53:11.217742 (+ 14878us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.217753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.218298 (+ 545us) server_negotiation.cc:407] Connection header received
0927 22:53:11.218382 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.218387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.218487 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.218615 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:11.219183 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.220427 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.222186 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.222938 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.223110 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.223186 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.223512 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.223637 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14690,"thread_start_us":15316,"threads_started":1}
I20260927 22:53:11.224399 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.202835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:41125 (local address 127.8.204.129:43489)
0927 22:53:11.217703 (+ 14868us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:11.217732 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:11.217964 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:11.218632 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:11.218636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:11.218662 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:11.218993 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.219001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.220577 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.220582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.222034 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.222046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.223158 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.223165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.223361 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.223383 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:53:11.223489 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:53:11.223616 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14664,"mutex_wait_us":44,"spinlock_wait_cycles":37120,"thread_start_us":15264,"threads_started":1}
I20260927 22:53:11.224509 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2700c7b15b45389962cdb8b021028d" candidate_uuid: "9e9987f4c9e0403d824db44b0c859525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" is_pre_election: true
I20260927 22:53:11.224820 11185 raft_consensus.cc:2454] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e9987f4c9e0403d824db44b0c859525 in term 0.
I20260927 22:53:11.225066 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2700c7b15b45389962cdb8b021028d" candidate_uuid: "9e9987f4c9e0403d824db44b0c859525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6" is_pre_election: true
I20260927 22:53:11.225318 11045 raft_consensus.cc:2454] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e9987f4c9e0403d824db44b0c859525 in term 0.
I20260927 22:53:11.225396 10837 leader_election.cc:302] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e9987f4c9e0403d824db44b0c859525, a462ceabf15b466b9e13bbb5dd5fe347; no voters:
I20260927 22:53:11.225837 11345 raft_consensus.cc:2781] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:11.225973 11345 raft_consensus.cc:482] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:11.226109 11345 raft_consensus.cc:3037] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:11.228343 11345 raft_consensus.cc:504] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.228976 11345 leader_election.cc:288] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [CANDIDATE]: Term 1 election: Requested vote from peers feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125), a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509)
I20260927 22:53:11.229395 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2700c7b15b45389962cdb8b021028d" candidate_uuid: "9e9987f4c9e0403d824db44b0c859525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb7399116844192a4a38bf6d102cbf6"
I20260927 22:53:11.229501 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc2700c7b15b45389962cdb8b021028d" candidate_uuid: "9e9987f4c9e0403d824db44b0c859525" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a462ceabf15b466b9e13bbb5dd5fe347"
I20260927 22:53:11.229590 11045 raft_consensus.cc:3037] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:11.229707 11185 raft_consensus.cc:3037] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:11.233053 11045 raft_consensus.cc:2454] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9987f4c9e0403d824db44b0c859525 in term 1.
I20260927 22:53:11.233089 11185 raft_consensus.cc:2454] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e9987f4c9e0403d824db44b0c859525 in term 1.
I20260927 22:53:11.233443 10840 leader_election.cc:302] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e9987f4c9e0403d824db44b0c859525, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:11.233740 11345 raft_consensus.cc:2781] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:11.234182 11345 raft_consensus.cc:686] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 1 LEADER]: Becoming Leader. State: Replica: 9e9987f4c9e0403d824db44b0c859525, State: Running, Role: LEADER
I20260927 22:53:11.234575 11345 consensus_queue.cc:237] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.237542 10750 catalog_manager.cc:5495] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129). New cstate: current_term: 1 leader_uuid: "9e9987f4c9e0403d824db44b0c859525" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } health_report { overall_health: UNKNOWN } } }
I20260927 22:53:11.286128 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.279645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.8.204.129:43495)
0927 22:53:11.280174 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.280178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.280197 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:53:11.280275 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.280278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.280346 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.280519 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:11.280910 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.281777 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.283324 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.284194 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.284314 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.284840 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.285490 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.285623 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.285771 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.285841 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":315,"threads_started":1}
I20260927 22:53:11.300688 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.295912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:57437 (local address 127.8.204.131:39509)
0927 22:53:11.296089 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.296094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.296487 (+ 393us) server_negotiation.cc:407] Connection header received
0927 22:53:11.296623 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.296627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.296712 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.296793 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:11.297283 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.297952 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.299322 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.299725 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.299822 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.300181 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.300270 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.300331 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.300438 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.300495 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:53:11.300768 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.295760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.8.204.129:57437)
0927 22:53:11.296365 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:11.296391 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:11.296514 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:11.296866 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:11.296870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:11.296880 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:11.297138 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.297145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.298067 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.298070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.299149 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.299160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.299875 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.299880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.299995 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.300044 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.300346 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.300353 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:53:11.300410 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 22:53:11.300509 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":297,"threads_started":1}
I20260927 22:53:11.308861 11185 raft_consensus.cc:1261] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Refusing update from remote peer 9e9987f4c9e0403d824db44b0c859525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:11.309442 11045 raft_consensus.cc:1261] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Refusing update from remote peer 9e9987f4c9e0403d824db44b0c859525: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:11.309515 11351 consensus_queue.cc:1035] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:11.309981 11345 consensus_queue.cc:1035] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:11.334408 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.328472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:55485 (local address 127.8.204.129:43495)
0927 22:53:11.328649 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.328656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.329271 (+ 615us) server_negotiation.cc:407] Connection header received
0927 22:53:11.329457 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.329462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.329550 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.329632 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:11.330099 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.330902 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.332415 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.333233 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.333368 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.333745 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.333862 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.333949 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.334059 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.334129 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:53:11.334554 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.328575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:43495 (local address 127.8.204.131:55485)
0927 22:53:11.329183 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:11.329197 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:11.329336 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:11.329657 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:11.329661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:11.329675 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:11.329950 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.329958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.331030 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.331037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.332285 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.332297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.333399 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.333406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.333563 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.333596 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.333950 (+ 354us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.333962 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:53:11.334039 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:53:11.334164 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":351,"threads_started":1}
I20260927 22:53:11.364259 10750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:11.364522 10750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:11.367564 10750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Sending DeleteTablet for 3 replicas of tablet cc2700c7b15b45389962cdb8b021028d
I20260927 22:53:11.368269 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet cc2700c7b15b45389962cdb8b021028d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:11 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:47468
I20260927 22:53:11.368426 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet cc2700c7b15b45389962cdb8b021028d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:11 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49070
I20260927 22:53:11.368580 11328 tablet_replica.cc:330] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: stopping tablet replica
I20260927 22:53:11.368667 11330 tablet_replica.cc:330] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: stopping tablet replica
I20260927 22:53:11.368778 11328 raft_consensus.cc:2229] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:11.368849 11330 raft_consensus.cc:2229] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:11.369036 11330 raft_consensus.cc:2258] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:11.369280 11328 raft_consensus.cc:2258] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:11.369691 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet cc2700c7b15b45389962cdb8b021028d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:11 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:60054
I20260927 22:53:11.370185 11330 ts_tablet_manager.cc:1905] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:11.370343 11328 ts_tablet_manager.cc:1905] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:11.370357 10750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:11.371325 10750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:11.372176 11329 tablet_replica.cc:330] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: stopping tablet replica
I20260927 22:53:11.372378 11329 raft_consensus.cc:2229] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:11.372584 11329 raft_consensus.cc:2258] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:11.374768 11329 ts_tablet_manager.cc:1905] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:11.377446 11330 ts_tablet_manager.cc:1918] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:11.377576 11330 log.cc:1196] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/cc2700c7b15b45389962cdb8b021028d
I20260927 22:53:11.377974 11330 ts_tablet_manager.cc:1939] T cc2700c7b15b45389962cdb8b021028d P feb7399116844192a4a38bf6d102cbf6: Deleting consensus metadata
I20260927 22:53:11.378916 10887 tablet_service.cc:1467] Processing CreateTablet for tablet f0260ff423d3498392b2b2dfd873363c (DEFAULT_TABLE table=test-table [id=b538fc24231e46f098527035be3bdabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:11.379906 11026 tablet_service.cc:1467] Processing CreateTablet for tablet f0260ff423d3498392b2b2dfd873363c (DEFAULT_TABLE table=test-table [id=b538fc24231e46f098527035be3bdabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:11.380503 11165 tablet_service.cc:1467] Processing CreateTablet for tablet f0260ff423d3498392b2b2dfd873363c (DEFAULT_TABLE table=test-table [id=b538fc24231e46f098527035be3bdabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:11.380748 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0260ff423d3498392b2b2dfd873363c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:11.380923 11165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0260ff423d3498392b2b2dfd873363c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:11.381930 11329 ts_tablet_manager.cc:1918] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:11.382082 11329 log.cc:1196] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/cc2700c7b15b45389962cdb8b021028d
I20260927 22:53:11.382503 11329 ts_tablet_manager.cc:1939] T cc2700c7b15b45389962cdb8b021028d P a462ceabf15b466b9e13bbb5dd5fe347: Deleting consensus metadata
I20260927 22:53:11.383431 10737 catalog_manager.cc:4841] TS feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125): tablet cc2700c7b15b45389962cdb8b021028d (table test-table [id=0da170f2603e4610a86d6223617d57f9]) successfully deleted
I20260927 22:53:11.383761 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0260ff423d3498392b2b2dfd873363c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:11.386933 11341 tablet_bootstrap.cc:492] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Bootstrap starting.
I20260927 22:53:11.387209 11328 ts_tablet_manager.cc:1918] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:11.387387 11328 log.cc:1196] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/cc2700c7b15b45389962cdb8b021028d
I20260927 22:53:11.388132 11328 ts_tablet_manager.cc:1939] T cc2700c7b15b45389962cdb8b021028d P 9e9987f4c9e0403d824db44b0c859525: Deleting consensus metadata
I20260927 22:53:11.388890 11341 tablet_bootstrap.cc:654] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:11.389008 10737 catalog_manager.cc:4841] TS 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495): tablet cc2700c7b15b45389962cdb8b021028d (table test-table [id=0da170f2603e4610a86d6223617d57f9]) successfully deleted
I20260927 22:53:11.389894 10734 catalog_manager.cc:4841] TS a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509): tablet cc2700c7b15b45389962cdb8b021028d (table test-table [id=0da170f2603e4610a86d6223617d57f9]) successfully deleted
I20260927 22:53:11.391075 11341 tablet_bootstrap.cc:492] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: No bootstrap required, opened a new log
I20260927 22:53:11.391225 11341 ts_tablet_manager.cc:1397] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:53:11.391868 11339 tablet_bootstrap.cc:492] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Bootstrap starting.
I20260927 22:53:11.391983 11341 raft_consensus.cc:348] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.392216 11341 raft_consensus.cc:374] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:11.392302 11341 raft_consensus.cc:729] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a462ceabf15b466b9e13bbb5dd5fe347, State: Initialized, Role: FOLLOWER
I20260927 22:53:11.392550 11341 consensus_queue.cc:260] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.393471 11339 tablet_bootstrap.cc:654] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:11.395165 11341 ts_tablet_manager.cc:1428] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:53:11.397758 11340 tablet_bootstrap.cc:492] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Bootstrap starting.
I20260927 22:53:11.399230 11340 tablet_bootstrap.cc:654] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:11.399462 11339 tablet_bootstrap.cc:492] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: No bootstrap required, opened a new log
I20260927 22:53:11.399638 11339 ts_tablet_manager.cc:1397] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260927 22:53:11.400339 11339 raft_consensus.cc:348] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.400547 11339 raft_consensus.cc:374] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:11.400815 11339 raft_consensus.cc:729] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e9987f4c9e0403d824db44b0c859525, State: Initialized, Role: FOLLOWER
I20260927 22:53:11.401192 11340 tablet_bootstrap.cc:492] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: No bootstrap required, opened a new log
I20260927 22:53:11.401100 11339 consensus_queue.cc:260] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.401695 11339 ts_tablet_manager.cc:1428] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:11.401847 11340 ts_tablet_manager.cc:1397] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 22:53:11.402458 11340 raft_consensus.cc:348] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.402662 11340 raft_consensus.cc:374] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:11.402752 11340 raft_consensus.cc:729] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb7399116844192a4a38bf6d102cbf6, State: Initialized, Role: FOLLOWER
I20260927 22:53:11.402987 11340 consensus_queue.cc:260] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.403590 11340 ts_tablet_manager.cc:1428] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:53:11.510654 11346 raft_consensus.cc:482] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:11.510905 11346 raft_consensus.cc:504] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.511656 11346 leader_election.cc:288] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509)
I20260927 22:53:11.512212 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0260ff423d3498392b2b2dfd873363c" candidate_uuid: "feb7399116844192a4a38bf6d102cbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" is_pre_election: true
I20260927 22:53:11.512472 11185 raft_consensus.cc:2454] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb7399116844192a4a38bf6d102cbf6 in term 0.
I20260927 22:53:11.512650 10906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0260ff423d3498392b2b2dfd873363c" candidate_uuid: "feb7399116844192a4a38bf6d102cbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525" is_pre_election: true
I20260927 22:53:11.512903 10906 raft_consensus.cc:2454] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb7399116844192a4a38bf6d102cbf6 in term 0.
I20260927 22:53:11.512908 10977 leader_election.cc:302] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a462ceabf15b466b9e13bbb5dd5fe347, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:11.513201 11346 raft_consensus.cc:2781] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:11.513336 11346 raft_consensus.cc:482] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:11.513434 11346 raft_consensus.cc:3037] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:11.516530 11346 raft_consensus.cc:504] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.517144 11346 leader_election.cc:288] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [CANDIDATE]: Term 1 election: Requested vote from peers 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495), a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509)
I20260927 22:53:11.517620 11185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0260ff423d3498392b2b2dfd873363c" candidate_uuid: "feb7399116844192a4a38bf6d102cbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a462ceabf15b466b9e13bbb5dd5fe347"
I20260927 22:53:11.517604 10906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0260ff423d3498392b2b2dfd873363c" candidate_uuid: "feb7399116844192a4a38bf6d102cbf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e9987f4c9e0403d824db44b0c859525"
I20260927 22:53:11.517831 10906 raft_consensus.cc:3037] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:11.517845 11185 raft_consensus.cc:3037] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:11.520706 11185 raft_consensus.cc:2454] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb7399116844192a4a38bf6d102cbf6 in term 1.
I20260927 22:53:11.520861 10906 raft_consensus.cc:2454] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb7399116844192a4a38bf6d102cbf6 in term 1.
I20260927 22:53:11.521183 10977 leader_election.cc:302] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a462ceabf15b466b9e13bbb5dd5fe347, feb7399116844192a4a38bf6d102cbf6; no voters:
I20260927 22:53:11.521443 11346 raft_consensus.cc:2781] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:11.521916 11346 raft_consensus.cc:686] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 1 LEADER]: Becoming Leader. State: Replica: feb7399116844192a4a38bf6d102cbf6, State: Running, Role: LEADER
I20260927 22:53:11.522516 11346 consensus_queue.cc:237] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } }
I20260927 22:53:11.526050 10749 catalog_manager.cc:5495] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb7399116844192a4a38bf6d102cbf6 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "feb7399116844192a4a38bf6d102cbf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb7399116844192a4a38bf6d102cbf6" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 41125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 } health_report { overall_health: UNKNOWN } } }
I20260927 22:53:11.569810 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.556927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.8.204.130:41125)
0927 22:53:11.557102 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.557108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.557126 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:53:11.557218 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.557223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.557318 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.557480 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:11.557893 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.559359 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.561293 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.567071 (+ 5778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.567212 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.567862 (+ 650us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.568646 (+ 784us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.568803 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.569432 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.569516 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 22:53:11.611585 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.592511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:47907 (local address 127.8.204.131:39509)
0927 22:53:11.592698 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.592704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.605193 (+ 12489us) server_negotiation.cc:407] Connection header received
0927 22:53:11.605390 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.605395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.605499 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.605609 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:11.606311 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.607499 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.609965 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.610344 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.610436 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.610944 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.611038 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.611103 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.611244 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.611364 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:53:11.611882 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.592091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39509 (local address 127.8.204.130:47907)
0927 22:53:11.605042 (+ 12951us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:11.605066 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:11.605241 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:11.605775 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:11.605781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:11.605796 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:11.606150 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.606163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.607656 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.607662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.609768 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.609784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.610521 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.610528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.610712 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.610783 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.611125 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.611138 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:53:11.611233 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 22:53:11.611393 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12744,"thread_start_us":12597,"threads_started":1}
I20260927 22:53:11.620500 10906 raft_consensus.cc:1261] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Refusing update from remote peer feb7399116844192a4a38bf6d102cbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:11.620728 11185 raft_consensus.cc:1261] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Refusing update from remote peer feb7399116844192a4a38bf6d102cbf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:11.621079 11367 consensus_queue.cc:1035] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e9987f4c9e0403d824db44b0c859525" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 43495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:11.621497 11346 consensus_queue.cc:1035] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a462ceabf15b466b9e13bbb5dd5fe347" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:11.649860 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.643669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:50467 (local address 127.8.204.130:41125)
0927 22:53:11.643819 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:11.643824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:11.643843 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:53:11.644001 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:11.644006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:11.644098 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:11.644177 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:11.644532 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.645656 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.648034 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.648576 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.648706 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.649136 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.649227 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.649292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:11.649441 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 22:53:11.649549 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:11.649897 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:11.643463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:41125 (local address 127.8.204.131:50467)
0927 22:53:11.643698 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:11.643714 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:11.643835 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:11.644192 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:11.644197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:11.644209 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:11.644374 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.644389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.645805 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.645811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.647437 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:11.647457 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:11.648736 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:11.648743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:11.648934 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:11.648994 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:11.649316 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:11.649328 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:53:11.649433 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:53:11.649600 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 22:53:11.651198 11312 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:53:13.247434 10750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:13.247709 10750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:13.252048 10750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360e06bb6de24c449b5cebdfc4a6f113: Sending DeleteTablet for 3 replicas of tablet f0260ff423d3498392b2b2dfd873363c
I20260927 22:53:13.252755 10887 tablet_service.cc:1514] Processing DeleteTablet for tablet f0260ff423d3498392b2b2dfd873363c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:13 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:47468
I20260927 22:53:13.252774 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet f0260ff423d3498392b2b2dfd873363c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:13 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49070
I20260927 22:53:13.252848 11165 tablet_service.cc:1514] Processing DeleteTablet for tablet f0260ff423d3498392b2b2dfd873363c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:13 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:60054
I20260927 22:53:13.253433 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10821
I20260927 22:53:13.264401 11403 tablet_replica.cc:330] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: stopping tablet replica
I20260927 22:53:13.264688 11403 raft_consensus.cc:2229] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:13.265331 11403 raft_consensus.cc:2258] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:13.266070 11402 tablet_replica.cc:330] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: stopping tablet replica
I20260927 22:53:13.266327 11402 raft_consensus.cc:2229] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:13.266623 11402 raft_consensus.cc:2258] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:13.266903 11403 ts_tablet_manager.cc:1905] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:13.267907 11402 ts_tablet_manager.cc:1905] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:13.272441 11404 tablet_replica.cc:330] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: stopping tablet replica
I20260927 22:53:13.272682 11404 raft_consensus.cc:2229] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:13.273015 11404 raft_consensus.cc:2258] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:53:13 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:13.274047 11404 ts_tablet_manager.cc:1905] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:13.275238 11403 ts_tablet_manager.cc:1918] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:53:13.275481 11403 log.cc:1196] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/f0260ff423d3498392b2b2dfd873363c
I20260927 22:53:13.275977 11403 ts_tablet_manager.cc:1939] T f0260ff423d3498392b2b2dfd873363c P feb7399116844192a4a38bf6d102cbf6: Deleting consensus metadata
I20260927 22:53:13.276553 11402 ts_tablet_manager.cc:1918] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:53:13.276705 11402 log.cc:1196] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/f0260ff423d3498392b2b2dfd873363c
I20260927 22:53:13.277236 10737 catalog_manager.cc:4841] TS feb7399116844192a4a38bf6d102cbf6 (127.8.204.130:41125): tablet f0260ff423d3498392b2b2dfd873363c (table test-table [id=b538fc24231e46f098527035be3bdabc]) successfully deleted
I20260927 22:53:13.277287 11402 ts_tablet_manager.cc:1939] T f0260ff423d3498392b2b2dfd873363c P 9e9987f4c9e0403d824db44b0c859525: Deleting consensus metadata
I20260927 22:53:13.278687 10737 catalog_manager.cc:4841] TS 9e9987f4c9e0403d824db44b0c859525 (127.8.204.129:43495): tablet f0260ff423d3498392b2b2dfd873363c (table test-table [id=b538fc24231e46f098527035be3bdabc]) successfully deleted
I20260927 22:53:13.281492 11404 ts_tablet_manager.cc:1918] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:53:13.281641 11404 log.cc:1196] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/f0260ff423d3498392b2b2dfd873363c
I20260927 22:53:13.282141 11404 ts_tablet_manager.cc:1939] T f0260ff423d3498392b2b2dfd873363c P a462ceabf15b466b9e13bbb5dd5fe347: Deleting consensus metadata
I20260927 22:53:13.282526 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:13.255677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54332 (local address 127.8.204.129:43495)
0927 22:53:13.261795 (+ 6118us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:13.261804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:13.261828 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:13.261898 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:13.261906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:13.262016 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:13.262141 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:13.264222 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:13.265932 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:13.268070 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:13.268724 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:13.268844 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:13.274319 (+ 5475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:13.274337 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:13.274358 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:13.274403 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:13.279444 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:13.280320 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:13.280327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:13.280329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:13.280507 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:13.280959 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:13.280962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:13.280964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:13.281288 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:13.281648 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:13.281971 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 22:53:13.282127 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5992,"thread_start_us":5907,"threads_started":1}
W20260927 22:53:13.283172 10952 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54332
I20260927 22:53:13.283246 10734 catalog_manager.cc:4841] TS a462ceabf15b466b9e13bbb5dd5fe347 (127.8.204.131:39509): tablet f0260ff423d3498392b2b2dfd873363c (table test-table [id=b538fc24231e46f098527035be3bdabc]) successfully deleted
W20260927 22:53:13.283890 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10821: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:13.284125 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10821
I20260927 22:53:13.306950 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10961
Sep 27 22:53:13 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:13.321815 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:13.307810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.8.204.130:41125)
0927 22:53:13.308265 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:13.308270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:13.308284 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:53:13.308333 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:13.308337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:13.308405 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:13.308526 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:13.309745 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:13.310449 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:13.311768 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:13.312185 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:13.312317 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:13.316318 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:13.316332 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:13.316347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:13.316383 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:13.319124 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:13.319942 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:13.319950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:13.319952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:13.320103 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:13.320488 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:13.320492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:13.320494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:13.320804 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:13.321044 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:13.321297 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:53:13.321464 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":235,"threads_started":1}
W20260927 22:53:13.322505 11091 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45384
W20260927 22:53:13.323150 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10961: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:13.323335 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10961
I20260927 22:53:13.346901 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11100
Sep 27 22:53:13 dist-test-slave-40vx krb5kdc[10705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549591, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:13.361366 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:13.347645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59626 (local address 127.8.204.131:39509)
0927 22:53:13.348161 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:13.348165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:13.348182 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:53:13.348233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:13.348237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:13.348293 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:13.348388 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:13.349711 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:13.350446 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:13.351713 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:13.352094 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:13.352186 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:13.356235 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:13.356249 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:13.356265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:13.356302 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:13.358883 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:13.359649 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:13.359657 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:13.359659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:13.359779 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:13.360081 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:13.360084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:13.360086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:13.360409 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:13.360645 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:13.360919 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:53:13.361094 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":271,"threads_started":1}
W20260927 22:53:13.361995 11230 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59626
W20260927 22:53:13.362596 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11100: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:13.362764 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 11100
I20260927 22:53:13.386906 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 10718
I20260927 22:53:13.387981 10790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:13.503472 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 10718
2026-09-27T22:53:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8254 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:13 dist-test-slave-40vx krb5kdc[11412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:13 dist-test-slave-40vx krb5kdc[11412](info): set up 2 sockets
Sep 27 22:53:13 dist-test-slave-40vx krb5kdc[11412](info): commencing operation
krb5kdc: starting...
W20260927 22:53:14.583590 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:14 dist-test-slave-40vx krb5kdc[11412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:14.767570 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:41055
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:41055
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:15.080343 11425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:15.080642 11425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:15.080821 11425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:15.088136 11425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:15.088251 11425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:15.088346 11425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:15.088430 11425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:15.088515 11425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:15.098389 11425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:41055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:41055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:15.099814 11425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:15.101254 11425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:15.113107 11433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:15.113236 11431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:15.113510 11430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:15.114437 11425 server_base.cc:1034] running on GCE node
I20260927 22:53:15.115242 11425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:15.116771 11425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:15.118019 11425 hybrid_clock.cc:648] HybridClock initialized: now 1790549595117959 us; error 68 us; skew 500 ppm
I20260927 22:53:15.118378 11425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:15 dist-test-slave-40vx krb5kdc[11412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549595, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:15.121381 11425 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:15.122746 11425 webserver.cc:460] Webserver started at http://127.8.204.190:44601/ using document root <none> and password file <none>
I20260927 22:53:15.123507 11425 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:15.123646 11425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:15.123958 11425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:15.126798 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "02219813a4bb4fde90d050f45330b466"
format_stamp: "Formatted at 2026-09-27 22:53:15 on dist-test-slave-40vx"
server_key: "84db47e00c3aa13c65c179486356aa5d"
server_key_iv: "280e9c8d874a725790786e4731bcb17a"
server_key_version: "encryptionkey@0"
I20260927 22:53:15.127629 11425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "02219813a4bb4fde90d050f45330b466"
format_stamp: "Formatted at 2026-09-27 22:53:15 on dist-test-slave-40vx"
server_key: "84db47e00c3aa13c65c179486356aa5d"
server_key_iv: "280e9c8d874a725790786e4731bcb17a"
server_key_version: "encryptionkey@0"
I20260927 22:53:15.132866 11425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:53:15.136302 11440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:15.137555 11425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:53:15.137724 11425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "02219813a4bb4fde90d050f45330b466"
format_stamp: "Formatted at 2026-09-27 22:53:15 on dist-test-slave-40vx"
server_key: "84db47e00c3aa13c65c179486356aa5d"
server_key_iv: "280e9c8d874a725790786e4731bcb17a"
server_key_version: "encryptionkey@0"
I20260927 22:53:15.137925 11425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:15.164193 11425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:15.168041 11425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:15.168324 11425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:15.186435 11425 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:41055
I20260927 22:53:15.186482 11491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:41055 every 8 connection(s)
I20260927 22:53:15.187808 11425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:15.191496 11492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:15.194008 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 11425
I20260927 22:53:15.194226 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:15.195024 9010 external_mini_cluster.cc:1428] Setting key aef16dca26108b164feb5362497c8077
I20260927 22:53:15.204910 11492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466: Bootstrap starting.
I20260927 22:53:15.208324 11492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:15.209282 11492 log.cc:824] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:53:15 dist-test-slave-40vx krb5kdc[11412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:15.212181 11492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466: No bootstrap required, opened a new log
I20260927 22:53:15.219413 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:15.197421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.8.204.190:41055)
0927 22:53:15.198321 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:15.198336 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:15.198791 (+ 455us) server_negotiation.cc:407] Connection header received
0927 22:53:15.200729 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:15.200756 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:15.201264 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:15.201699 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:15.202897 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:15.204857 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:15.206090 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:15.207190 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:15.207397 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:15.211229 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:15.211242 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:15.211260 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:15.211331 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:15.215213 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:15.215876 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:15.215884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:15.215892 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:15.216099 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:15.216428 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:15.216433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:15.216445 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:15.217089 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:15.217371 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:15.217730 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 22:53:15.218130 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":422,"threads_started":1}
I20260927 22:53:15.221922 11492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02219813a4bb4fde90d050f45330b466" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41055 } }
I20260927 22:53:15.222268 11492 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:15.222360 11492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02219813a4bb4fde90d050f45330b466, State: Initialized, Role: FOLLOWER
I20260927 22:53:15.222889 11492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02219813a4bb4fde90d050f45330b466" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41055 } }
I20260927 22:53:15.223110 11492 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:15.223228 11492 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:15.223393 11492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:15.225292 11492 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02219813a4bb4fde90d050f45330b466" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41055 } }
I20260927 22:53:15.225730 11492 leader_election.cc:302] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02219813a4bb4fde90d050f45330b466; no voters:
I20260927 22:53:15.226294 11492 leader_election.cc:288] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:15.226441 11497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:15.227066 11497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [term 1 LEADER]: Becoming Leader. State: Replica: 02219813a4bb4fde90d050f45330b466, State: Running, Role: LEADER
I20260927 22:53:15.227684 11497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02219813a4bb4fde90d050f45330b466" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41055 } }
I20260927 22:53:15.229485 11492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:15.230281 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02219813a4bb4fde90d050f45330b466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02219813a4bb4fde90d050f45330b466" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41055 } } }
I20260927 22:53:15.230294 11498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02219813a4bb4fde90d050f45330b466. Latest consensus state: current_term: 1 leader_uuid: "02219813a4bb4fde90d050f45330b466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02219813a4bb4fde90d050f45330b466" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41055 } } }
I20260927 22:53:15.230690 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:15.232264 11498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:15.236804 11502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:15.242069 11502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:15.250741 11502 catalog_manager.cc:1347] Generated new cluster ID: 2acd059575de4ca8a4d2b2a0054c11f1
I20260927 22:53:15.250849 11502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:15.283995 11502 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:15.285352 11502 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:15.297881 11502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02219813a4bb4fde90d050f45330b466: Generated new TSK 0
I20260927 22:53:15.298779 11502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:15.346336 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41055
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:15.672459 11518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:15.672750 11518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:15.672883 11518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:15.680122 11518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:15.680238 11518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:15.680397 11518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:15.689980 11518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:15.691262 11518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:15.692565 11518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:15.705791 11524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:15.705864 11523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:15.705864 11526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:15.706652 11518 server_base.cc:1034] running on GCE node
I20260927 22:53:15.707258 11518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:15.708268 11518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:15.709481 11518 hybrid_clock.cc:648] HybridClock initialized: now 1790549595709440 us; error 42 us; skew 500 ppm
I20260927 22:53:15.709821 11518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:15 dist-test-slave-40vx krb5kdc[11412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549595, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:15.717327 11518 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:15.718956 11518 webserver.cc:460] Webserver started at http://127.8.204.129:41897/ using document root <none> and password file <none>
I20260927 22:53:15.719726 11518 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:15.719852 11518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:15.720173 11518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:15.723049 11518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "acdb9ae6c7b146e78f241a453c375f1d"
format_stamp: "Formatted at 2026-09-27 22:53:15 on dist-test-slave-40vx"
server_key: "042a47471a4eb5ab4ce8821c2741037d"
server_key_iv: "6dbaad633c4aa402ad09fb4037deb6d4"
server_key_version: "encryptionkey@0"
I20260927 22:53:15.723909 11518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "acdb9ae6c7b146e78f241a453c375f1d"
format_stamp: "Formatted at 2026-09-27 22:53:15 on dist-test-slave-40vx"
server_key: "042a47471a4eb5ab4ce8821c2741037d"
server_key_iv: "6dbaad633c4aa402ad09fb4037deb6d4"
server_key_version: "encryptionkey@0"
I20260927 22:53:15.729000 11518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:53:15.732743 11533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:15.734088 11518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:15.734254 11518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "acdb9ae6c7b146e78f241a453c375f1d"
format_stamp: "Formatted at 2026-09-27 22:53:15 on dist-test-slave-40vx"
server_key: "042a47471a4eb5ab4ce8821c2741037d"
server_key_iv: "6dbaad633c4aa402ad09fb4037deb6d4"
server_key_version: "encryptionkey@0"
I20260927 22:53:15.734447 11518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:15.761555 11518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:15.765878 11518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:15.766167 11518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:15.767114 11518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:15.768467 11518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:15.768599 11518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:15.768725 11518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:15.768812 11518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:15.800694 11518 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:37013
I20260927 22:53:15.800774 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:37013 every 8 connection(s)
I20260927 22:53:15.801900 11518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:15.805266 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 11518
I20260927 22:53:15.805491 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:15.806509 9010 external_mini_cluster.cc:1428] Setting key 2e006d6d30649f8166c2a8360d6b2957
Sep 27 22:53:15 dist-test-slave-40vx krb5kdc[11412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549595, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:15.824775 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:15.804322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:38829 (local address 127.8.204.190:41055)
0927 22:53:15.804760 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:15.804764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:15.805703 (+ 939us) server_negotiation.cc:407] Connection header received
0927 22:53:15.808096 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:15.808104 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:15.808264 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:15.808385 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:15.810017 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:15.811128 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:15.813669 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:15.814064 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:15.814166 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:15.818000 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:15.818009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:15.818017 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:15.818063 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:15.821548 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:15.822245 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:15.822253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:15.822260 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:15.822407 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:15.822832 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:15.822840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:15.822845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:15.823218 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:15.823449 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:15.824082 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 22:53:15.824339 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":254,"threads_started":1}
I20260927 22:53:15.825395 11649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:15.804749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:41055 (local address 127.8.204.129:38829)
0927 22:53:15.805532 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:15.805583 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:15.807776 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:15.808685 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:15.808696 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:15.809301 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:15.809846 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:15.809865 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:15.811315 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:15.811324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:15.813508 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:15.813537 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:15.814290 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:15.814300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:15.814643 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:15.815433 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:15.815456 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:15.817771 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:15.821689 (+ 3918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:15.821696 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:15.821710 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:15.822113 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:15.822546 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:15.822551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:15.822553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:15.822712 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:15.823392 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:15.823398 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:15.823699 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 22:53:15.823915 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 22:53:15.824289 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":295,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:53:15.826994 11646 heartbeater.cc:347] Connected to a master server at 127.8.204.190:41055
I20260927 22:53:15.827365 11646 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:15.828096 11646 heartbeater.cc:511] Master 127.8.204.190:41055 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:53:15.830397 11457 ts_manager.cc:194] Registered new tserver with Master: acdb9ae6c7b146e78f241a453c375f1d (127.8.204.129:37013)
I20260927 22:53:15.832590 11457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:38829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:15.852324 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41055
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:16.167346 11652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:16.167661 11652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:16.167829 11652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:16.174897 11652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:16.175056 11652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:16.175225 11652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:16.184502 11652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:16.185770 11652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:16.187057 11652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:16.199703 11658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:16.199867 11657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:16.200230 11660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:16.200551 11652 server_base.cc:1034] running on GCE node
I20260927 22:53:16.201136 11652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:16.202102 11652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:16.203346 11652 hybrid_clock.cc:648] HybridClock initialized: now 1790549596203291 us; error 48 us; skew 500 ppm
I20260927 22:53:16.203593 11652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:16 dist-test-slave-40vx krb5kdc[11412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:16.206511 11652 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:16.207830 11652 webserver.cc:460] Webserver started at http://127.8.204.130:46521/ using document root <none> and password file <none>
I20260927 22:53:16.208436 11652 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:16.208524 11652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:16.208755 11652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:16.211463 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "934e76202d2144c79ec57b2b797600c7"
format_stamp: "Formatted at 2026-09-27 22:53:16 on dist-test-slave-40vx"
server_key: "bcead7855052278ddeb15e7ac792956e"
server_key_iv: "a68f9fa82d6d36a4f28dd73a718cd759"
server_key_version: "encryptionkey@0"
I20260927 22:53:16.212203 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "934e76202d2144c79ec57b2b797600c7"
format_stamp: "Formatted at 2026-09-27 22:53:16 on dist-test-slave-40vx"
server_key: "bcead7855052278ddeb15e7ac792956e"
server_key_iv: "a68f9fa82d6d36a4f28dd73a718cd759"
server_key_version: "encryptionkey@0"
I20260927 22:53:16.216895 11652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:53:16.220050 11667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:16.221297 11652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:16.221470 11652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "934e76202d2144c79ec57b2b797600c7"
format_stamp: "Formatted at 2026-09-27 22:53:16 on dist-test-slave-40vx"
server_key: "bcead7855052278ddeb15e7ac792956e"
server_key_iv: "a68f9fa82d6d36a4f28dd73a718cd759"
server_key_version: "encryptionkey@0"
I20260927 22:53:16.221648 11652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:16.243124 11652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:16.246948 11652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:16.247211 11652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:16.248116 11652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:16.249359 11652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:16.249439 11652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:16.249594 11652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:16.249652 11652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:16.281097 11652 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:44377
I20260927 22:53:16.281188 11779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:44377 every 8 connection(s)
I20260927 22:53:16.282352 11652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:16.288717 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 11652
I20260927 22:53:16.289052 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:16.289549 9010 external_mini_cluster.cc:1428] Setting key 96c0fdaf7a780da7f49b7450edb8bf44
Sep 27 22:53:16 dist-test-slave-40vx krb5kdc[11412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:16.301920 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:16.284917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:48009 (local address 127.8.204.190:41055)
0927 22:53:16.285095 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:16.285100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:16.286161 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 22:53:16.287905 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:16.287909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:16.287974 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:16.288064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:16.289592 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.290600 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.292221 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.292645 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.292778 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:16.296080 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:16.296085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:16.296088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:16.296119 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:16.298817 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:16.299601 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:16.299609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:16.299616 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:16.299727 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:16.300144 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:16.300149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:16.300152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:16.300423 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:16.300578 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:16.301366 (+ 788us) server_negotiation.cc:299] Negotiation successful
0927 22:53:16.301599 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:53:16.302850 11782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:16.285333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:41055 (local address 127.8.204.130:48009)
0927 22:53:16.285996 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:16.286042 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:16.287649 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:16.288253 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:16.288260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:16.288756 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:16.289407 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:16.289431 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.290754 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.290761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:16.292066 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:16.292078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.292860 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.292867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:16.293096 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:16.293765 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:16.293786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:16.295860 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:16.298961 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:16.298970 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:16.298985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:16.299482 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:16.299834 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:16.299840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:16.299844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:16.300030 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:16.300546 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:16.300555 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:16.300978 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 22:53:16.301272 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 22:53:16.301766 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
I20260927 22:53:16.304346 11780 heartbeater.cc:347] Connected to a master server at 127.8.204.190:41055
I20260927 22:53:16.304680 11780 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:16.305335 11780 heartbeater.cc:511] Master 127.8.204.190:41055 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:53:16.306818 11457 ts_manager.cc:194] Registered new tserver with Master: 934e76202d2144c79ec57b2b797600c7 (127.8.204.130:44377)
I20260927 22:53:16.307741 11457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:48009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:16.329967 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41055
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:16.642966 11785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:16.643262 11785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:16.643430 11785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:16.650908 11785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:16.651031 11785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:16.651142 11785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:16.661309 11785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41055
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:16.663100 11785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:16.664722 11785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:16.677706 11791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:16.677682 11793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:16.677654 11790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:16.678468 11785 server_base.cc:1034] running on GCE node
I20260927 22:53:16.679073 11785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:16.680094 11785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:16.681288 11785 hybrid_clock.cc:648] HybridClock initialized: now 1790549596681245 us; error 49 us; skew 500 ppm
I20260927 22:53:16.681607 11785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:16 dist-test-slave-40vx krb5kdc[11412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:16.684966 11785 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:16.686443 11785 webserver.cc:460] Webserver started at http://127.8.204.131:37297/ using document root <none> and password file <none>
I20260927 22:53:16.687152 11785 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:16.687336 11785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:16.687664 11785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:16.690582 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "fc1e2b08e0af40d994868e707a79a48c"
format_stamp: "Formatted at 2026-09-27 22:53:16 on dist-test-slave-40vx"
server_key: "2b79dd57244a944c528b7abed0cf4cdf"
server_key_iv: "e7c4717f9901d7556f3ee1a7eebd22c4"
server_key_version: "encryptionkey@0"
I20260927 22:53:16.691426 11785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "fc1e2b08e0af40d994868e707a79a48c"
format_stamp: "Formatted at 2026-09-27 22:53:16 on dist-test-slave-40vx"
server_key: "2b79dd57244a944c528b7abed0cf4cdf"
server_key_iv: "e7c4717f9901d7556f3ee1a7eebd22c4"
server_key_version: "encryptionkey@0"
I20260927 22:53:16.696550 11785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:53:16.699960 11800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:16.701241 11785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:16.701411 11785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "fc1e2b08e0af40d994868e707a79a48c"
format_stamp: "Formatted at 2026-09-27 22:53:16 on dist-test-slave-40vx"
server_key: "2b79dd57244a944c528b7abed0cf4cdf"
server_key_iv: "e7c4717f9901d7556f3ee1a7eebd22c4"
server_key_version: "encryptionkey@0"
I20260927 22:53:16.701623 11785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:16.715459 11785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:16.719691 11785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:16.719959 11785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:16.720847 11785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:16.722077 11785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:16.722174 11785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:16.722312 11785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:16.722384 11785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:16.755925 11785 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:35853
I20260927 22:53:16.756008 11912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:35853 every 8 connection(s)
I20260927 22:53:16.757170 11785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:16.767361 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 11785
I20260927 22:53:16.767575 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:16.768131 9010 external_mini_cluster.cc:1428] Setting key 0153f77d0e60be6678a15094fae566f5
Sep 27 22:53:16 dist-test-slave-40vx krb5kdc[11412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549596, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:16.776176 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:16.759920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:58501 (local address 127.8.204.190:41055)
0927 22:53:16.760098 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:16.760102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:16.761316 (+ 1214us) server_negotiation.cc:407] Connection header received
0927 22:53:16.763037 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:16.763041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:16.763108 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:16.763202 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:16.764688 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.765465 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.766967 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.767445 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.767575 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:16.771550 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:16.771554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:16.771557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:16.771586 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:16.773767 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:16.774368 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:16.774371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:16.774373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:16.774475 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:16.774815 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:16.774818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:16.774820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:16.775049 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:16.775151 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:16.775802 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 22:53:16.775946 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:53:16.777057 11915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:16.760465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:41055 (local address 127.8.204.131:58501)
0927 22:53:16.761155 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:16.761191 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:16.762756 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:16.763457 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:16.763465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:16.763950 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:16.764482 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:16.764498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.765601 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.765605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:16.766827 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:16.766838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:16.767699 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:16.767708 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:16.767997 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:16.768715 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:16.768735 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:16.771384 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:16.773888 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:16.773895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:16.773905 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:16.774272 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:16.774563 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:16.774566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:16.774568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:16.774729 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:16.775145 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:16.775150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:16.775482 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 22:53:16.775718 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 22:53:16.776056 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":266,"threads_started":1}
I20260927 22:53:16.778352 11913 heartbeater.cc:347] Connected to a master server at 127.8.204.190:41055
I20260927 22:53:16.778657 11913 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:16.779378 11913 heartbeater.cc:511] Master 127.8.204.190:41055 requested a full tablet report, sending...
I20260927 22:53:16.780645 11457 ts_manager.cc:194] Registered new tserver with Master: fc1e2b08e0af40d994868e707a79a48c (127.8.204.131:35853)
I20260927 22:53:16.781409 11457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:58501
I20260927 22:53:16.783181 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:53:16.812417 11648 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:16.809898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.8.204.190:41055)
0927 22:53:16.810122 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:16.810132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:16.810511 (+ 379us) server_negotiation.cc:407] Connection header received
0927 22:53:16.810657 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:16.810661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:16.810722 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:16.810810 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:16.812168 (+ 1358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49700: BlockingRecv error: recv got EOF from 127.0.0.1:49700 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:53:16.815801 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11518
W20260927 22:53:16.818653 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11518: Not authorized: Client connection negotiation failed: client connection to 127.8.204.129:37013: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:53:16.818848 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 11518
I20260927 22:53:16.835314 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11652
W20260927 22:53:16.837863 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11652: Not authorized: Client connection negotiation failed: client connection to 127.8.204.130:44377: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:53:16.838073 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 11652
I20260927 22:53:16.852854 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11785
W20260927 22:53:16.855863 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11785: Not authorized: Client connection negotiation failed: client connection to 127.8.204.131:35853: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:53:16.856029 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 11785
I20260927 22:53:16.871320 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11425
I20260927 22:53:16.872612 11487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:16.973768 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 11425
2026-09-27T22:53:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3467 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:17 dist-test-slave-40vx krb5kdc[11930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:17 dist-test-slave-40vx krb5kdc[11930](info): set up 2 sockets
Sep 27 22:53:17 dist-test-slave-40vx krb5kdc[11930](info): commencing operation
krb5kdc: starting...
W20260927 22:53:18.059607 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:18 dist-test-slave-40vx krb5kdc[11930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:18.236792 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45817
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:45817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:18.546188 11943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:18.546545 11943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:18.546721 11943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:18.553915 11943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:18.554006 11943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:18.554092 11943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:18.554191 11943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:18.554277 11943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:18.563896 11943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:45817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:18.565073 11943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:18.566318 11943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:18.577690 11948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:18.577713 11949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:18.577759 11951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:18.578572 11943 server_base.cc:1034] running on GCE node
I20260927 22:53:18.579210 11943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:18.580485 11943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:18.581670 11943 hybrid_clock.cc:648] HybridClock initialized: now 1790549598581621 us; error 52 us; skew 500 ppm
I20260927 22:53:18.581995 11943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:18 dist-test-slave-40vx krb5kdc[11930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549598, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:18.584864 11943 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:18.586251 11943 webserver.cc:460] Webserver started at http://127.8.204.190:33459/ using document root <none> and password file <none>
I20260927 22:53:18.586903 11943 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:18.587042 11943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:18.587404 11943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:18.590158 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "0433fa0a658e4c41ac557f4273b3114f"
format_stamp: "Formatted at 2026-09-27 22:53:18 on dist-test-slave-40vx"
server_key: "f20fc0660a54576aad4a429eccf1adb8"
server_key_iv: "efcec130658b9da6f4718185708db69f"
server_key_version: "encryptionkey@0"
I20260927 22:53:18.590914 11943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "0433fa0a658e4c41ac557f4273b3114f"
format_stamp: "Formatted at 2026-09-27 22:53:18 on dist-test-slave-40vx"
server_key: "f20fc0660a54576aad4a429eccf1adb8"
server_key_iv: "efcec130658b9da6f4718185708db69f"
server_key_version: "encryptionkey@0"
I20260927 22:53:18.595763 11943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260927 22:53:18.598856 11958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:18.600271 11943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:18.600459 11943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "0433fa0a658e4c41ac557f4273b3114f"
format_stamp: "Formatted at 2026-09-27 22:53:18 on dist-test-slave-40vx"
server_key: "f20fc0660a54576aad4a429eccf1adb8"
server_key_iv: "efcec130658b9da6f4718185708db69f"
server_key_version: "encryptionkey@0"
I20260927 22:53:18.600642 11943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:18.627239 11943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:18.630878 11943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:18.631165 11943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:18.649776 11943 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:45817
I20260927 22:53:18.649806 12009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:45817 every 8 connection(s)
I20260927 22:53:18.651237 11943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:18.652808 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 11943
I20260927 22:53:18.653036 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:18.653644 9010 external_mini_cluster.cc:1428] Setting key d825ea4c207e7d40876068b4e6db8792
I20260927 22:53:18.655476 12010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:53:18 dist-test-slave-40vx krb5kdc[11930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:18.669193 12010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f: Bootstrap starting.
I20260927 22:53:18.673399 12010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:18.674690 12010 log.cc:824] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:18.677775 12010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f: No bootstrap required, opened a new log
I20260927 22:53:18.678140 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:18.655535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.8.204.190:45817)
0927 22:53:18.656372 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:18.656389 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:18.656451 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:53:18.658548 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:18.658575 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:18.659127 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:18.659618 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:18.660995 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:18.663027 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:18.664318 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:18.665322 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:18.665537 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:18.669650 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:18.669660 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:18.669678 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:18.669731 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:18.673873 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:18.674540 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:18.674548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:18.674556 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:18.674752 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:18.675262 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:18.675269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:18.675274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:18.676114 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:18.676328 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:18.676790 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 22:53:18.677142 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":328,"threads_started":1}
I20260927 22:53:18.681703 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:53:18.685362 12010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0433fa0a658e4c41ac557f4273b3114f" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45817 } }
I20260927 22:53:18.685703 12010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:18.685801 12010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0433fa0a658e4c41ac557f4273b3114f, State: Initialized, Role: FOLLOWER
I20260927 22:53:18.686394 12010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0433fa0a658e4c41ac557f4273b3114f" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45817 } }
I20260927 22:53:18.686611 12010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:18.686735 12010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:18.686910 12010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:18.690497 12010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0433fa0a658e4c41ac557f4273b3114f" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45817 } }
I20260927 22:53:18.691156 12010 leader_election.cc:302] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0433fa0a658e4c41ac557f4273b3114f; no voters:
I20260927 22:53:18.691874 12010 leader_election.cc:288] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:18.691993 12016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:18.693531 12016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [term 1 LEADER]: Becoming Leader. State: Replica: 0433fa0a658e4c41ac557f4273b3114f, State: Running, Role: LEADER
I20260927 22:53:18.694108 12016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0433fa0a658e4c41ac557f4273b3114f" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45817 } }
I20260927 22:53:18.694320 12010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:18.697130 12017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0433fa0a658e4c41ac557f4273b3114f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0433fa0a658e4c41ac557f4273b3114f" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45817 } } }
I20260927 22:53:18.697108 12018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0433fa0a658e4c41ac557f4273b3114f. Latest consensus state: current_term: 1 leader_uuid: "0433fa0a658e4c41ac557f4273b3114f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0433fa0a658e4c41ac557f4273b3114f" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45817 } } }
I20260927 22:53:18.697517 12017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:18.697517 12018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:18.698551 12023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:18.711333 12023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:18.729988 12023 catalog_manager.cc:1347] Generated new cluster ID: cd9ad33603ba45fca9a9e58a783b3bed
I20260927 22:53:18.730108 12023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:18.748430 12023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:18.749984 12023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:18.759423 12023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0433fa0a658e4c41ac557f4273b3114f: Generated new TSK 0
I20260927 22:53:18.759601 12013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:18.749719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.8.204.190:45817)
0927 22:53:18.749875 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:18.749881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:18.750978 (+ 1097us) server_negotiation.cc:407] Connection header received
0927 22:53:18.751174 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:18.751181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:18.751262 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:18.751416 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:18.752615 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:18.753718 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:18.755613 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:18.756224 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:18.756364 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:18.757578 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:18.757585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:18.757593 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:18.758211 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:18.758285 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:18.758314 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:18.758432 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:18.758580 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:53:18.758691 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:18.760380 12023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:53:18.770999 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 11943
I20260927 22:53:18.772188 12005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:18.859575 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 11943
2026-09-27T22:53:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1880 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:18 dist-test-slave-40vx krb5kdc[12044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:18 dist-test-slave-40vx krb5kdc[12044](info): set up 2 sockets
Sep 27 22:53:18 dist-test-slave-40vx krb5kdc[12044](info): commencing operation
krb5kdc: starting...
W20260927 22:53:19.977792 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 22:53:20.177706 9010 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.200s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 22:53:20.460125 9010 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.282s user 0.001s sys 0.005s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 22:53:20.615738 9010 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.155s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/test-user.keytab.
W20260927 22:53:21.009555 9010 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.394s user 0.000s sys 0.004s
Sep 27 22:53:21 dist-test-slave-40vx krb5kdc[12044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:21.092970 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44317
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:21.402662 12057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:21.402941 12057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:21.403064 12057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:21.410037 12057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:21.410126 12057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:21.410233 12057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:21.410331 12057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:21.410408 12057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:21.419729 12057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:21.420934 12057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:21.422202 12057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:21.433219 12063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:21.433339 12065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:21.433380 12062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:21.434779 12057 server_base.cc:1034] running on GCE node
I20260927 22:53:21.435562 12057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:21.436780 12057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:21.437980 12057 hybrid_clock.cc:648] HybridClock initialized: now 1790549601437926 us; error 38 us; skew 500 ppm
I20260927 22:53:21.438269 12057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:21 dist-test-slave-40vx krb5kdc[12044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549601, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:21.446918 12057 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:21.448261 12057 webserver.cc:460] Webserver started at http://127.8.204.190:44619/ using document root <none> and password file <none>
I20260927 22:53:21.448932 12057 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:21.449026 12057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:21.449261 12057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:21.451988 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "4d6a17bc89e64a449f5a85610df30788"
format_stamp: "Formatted at 2026-09-27 22:53:21 on dist-test-slave-40vx"
server_key: "c51f51ad4ad570bc23112542bdd2ac4c"
server_key_iv: "44c0db9b761aa168cf6dca069c8cfd78"
server_key_version: "encryptionkey@0"
I20260927 22:53:21.452648 12057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "4d6a17bc89e64a449f5a85610df30788"
format_stamp: "Formatted at 2026-09-27 22:53:21 on dist-test-slave-40vx"
server_key: "c51f51ad4ad570bc23112542bdd2ac4c"
server_key_iv: "44c0db9b761aa168cf6dca069c8cfd78"
server_key_version: "encryptionkey@0"
I20260927 22:53:21.457221 12057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 22:53:21.460366 12072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:21.461531 12057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 22:53:21.461745 12057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "4d6a17bc89e64a449f5a85610df30788"
format_stamp: "Formatted at 2026-09-27 22:53:21 on dist-test-slave-40vx"
server_key: "c51f51ad4ad570bc23112542bdd2ac4c"
server_key_iv: "44c0db9b761aa168cf6dca069c8cfd78"
server_key_version: "encryptionkey@0"
I20260927 22:53:21.461915 12057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:21.476984 12057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:21.480628 12057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:21.480881 12057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:21.499053 12057 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:44317
I20260927 22:53:21.499074 12123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:44317 every 8 connection(s)
I20260927 22:53:21.500299 12057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:21.503540 12124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:21.509708 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12057
I20260927 22:53:21.509898 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:21.510495 9010 external_mini_cluster.cc:1428] Setting key ef357b8760ff5a96093b0f6897f88666
I20260927 22:53:21.513496 12124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788: Bootstrap starting.
I20260927 22:53:21.516966 12124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:21.518049 12124 log.cc:824] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:21.520732 12124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788: No bootstrap required, opened a new log
Sep 27 22:53:21 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:21.528460 12124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6a17bc89e64a449f5a85610df30788" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44317 } }
I20260927 22:53:21.528820 12124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:21.528959 12124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6a17bc89e64a449f5a85610df30788, State: Initialized, Role: FOLLOWER
I20260927 22:53:21.529577 12124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6a17bc89e64a449f5a85610df30788" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44317 } }
I20260927 22:53:21.529799 12124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:21.529943 12124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:21.530138 12124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:21.531920 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:21.512306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.8.204.190:44317)
0927 22:53:21.513062 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:21.513072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:21.513111 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:53:21.514811 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:21.514833 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:21.515223 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:21.515608 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:21.517349 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:21.518657 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:21.519977 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:21.520623 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:21.520813 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:21.524550 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:21.524557 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:21.524570 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:21.524614 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:21.527581 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:21.528224 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:21.528230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:21.528236 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:21.528381 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:21.528756 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:21.528760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:21.528762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:21.529384 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:21.529690 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:21.530091 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:53:21.530518 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":374,"threads_started":1}
I20260927 22:53:21.533056 12124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6a17bc89e64a449f5a85610df30788" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44317 } }
I20260927 22:53:21.533504 12124 leader_election.cc:302] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d6a17bc89e64a449f5a85610df30788; no voters:
I20260927 22:53:21.534081 12124 leader_election.cc:288] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:21.534209 12129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:21.535216 12129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [term 1 LEADER]: Becoming Leader. State: Replica: 4d6a17bc89e64a449f5a85610df30788, State: Running, Role: LEADER
I20260927 22:53:21.535753 12129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6a17bc89e64a449f5a85610df30788" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44317 } }
I20260927 22:53:21.536075 12124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:21.538303 12130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d6a17bc89e64a449f5a85610df30788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6a17bc89e64a449f5a85610df30788" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44317 } } }
I20260927 22:53:21.538726 12130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:21.538552 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d6a17bc89e64a449f5a85610df30788. Latest consensus state: current_term: 1 leader_uuid: "4d6a17bc89e64a449f5a85610df30788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6a17bc89e64a449f5a85610df30788" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44317 } } }
I20260927 22:53:21.539094 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:21.541569 12136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:21.546769 12136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:21.555189 12136 catalog_manager.cc:1347] Generated new cluster ID: 7898fa0fdca64754b1488ea3b1246541
I20260927 22:53:21.555311 12136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:21.569916 12136 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:21.571729 12136 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:21.591120 12136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d6a17bc89e64a449f5a85610df30788: Generated new TSK 0
I20260927 22:53:21.592269 12136 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:21.643314 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44317
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:21.967356 12150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:21.967612 12150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:21.967789 12150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:21.975358 12150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:21.975502 12150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:21.975687 12150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:21.985889 12150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:21.987126 12150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:21.988430 12150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:22.000929 12156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:22.000975 12158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:22.000906 12155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:22.001899 12150 server_base.cc:1034] running on GCE node
I20260927 22:53:22.002455 12150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:22.003436 12150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:22.004626 12150 hybrid_clock.cc:648] HybridClock initialized: now 1790549602004592 us; error 35 us; skew 500 ppm
I20260927 22:53:22.004946 12150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:22 dist-test-slave-40vx krb5kdc[12044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:22.008002 12150 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:22.009533 12150 webserver.cc:460] Webserver started at http://127.8.204.129:33875/ using document root <none> and password file <none>
I20260927 22:53:22.010265 12150 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:22.010396 12150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:22.010684 12150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:22.013500 12150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "4462a787f77c4d5283b4980de4b92856"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "c6f29be3fdf32e3810806a3539d19ebc"
server_key_iv: "aae2398e5c5c7d6182936aaf7b69a898"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.014254 12150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "4462a787f77c4d5283b4980de4b92856"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "c6f29be3fdf32e3810806a3539d19ebc"
server_key_iv: "aae2398e5c5c7d6182936aaf7b69a898"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.019106 12150 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:53:22.022598 12165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:22.024024 12150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:22.024196 12150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "4462a787f77c4d5283b4980de4b92856"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "c6f29be3fdf32e3810806a3539d19ebc"
server_key_iv: "aae2398e5c5c7d6182936aaf7b69a898"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.024402 12150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:22.059751 12150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:22.063616 12150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:22.063953 12150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:22.064836 12150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:22.066308 12150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:22.066391 12150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:22.066479 12150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:22.066527 12150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:22.099349 12150 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41509
I20260927 22:53:22.099398 12277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41509 every 8 connection(s)
I20260927 22:53:22.100630 12150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:22.102165 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12150
I20260927 22:53:22.102360 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:22.103086 9010 external_mini_cluster.cc:1428] Setting key ecd8b1c9d7d904123aaa401f13fbb496
Sep 27 22:53:22 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:53:22.122035 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:22.103325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:38277 (local address 127.8.204.190:44317)
0927 22:53:22.103795 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:22.103800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:22.104936 (+ 1136us) server_negotiation.cc:407] Connection header received
0927 22:53:22.106770 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:22.106775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:22.106874 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:22.106990 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:22.108501 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.109192 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.110705 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.111450 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.111597 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:22.115135 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:22.115143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:22.115151 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:22.115196 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:22.118826 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:22.119617 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:22.119623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:22.119626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:22.119765 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:22.120163 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:22.120168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:22.120172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:22.120555 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:22.120709 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:22.121363 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 22:53:22.121628 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":211,"threads_started":1}
I20260927 22:53:22.122776 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:22.103993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44317 (local address 127.8.204.129:38277)
0927 22:53:22.104767 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:22.104821 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:22.106502 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:22.107225 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:22.107233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:22.107781 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:22.108341 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:22.108358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.109350 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.109354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:22.110560 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:22.110571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.111661 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.111666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:22.111864 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:22.112588 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:22.112613 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:22.114903 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:22.118988 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:22.118995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:22.119007 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:22.119497 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:22.119872 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:22.119876 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:22.119878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:22.120042 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:22.120672 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:22.120678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:22.120985 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 22:53:22.121196 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 22:53:22.121575 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":293,"threads_started":1}
I20260927 22:53:22.124351 12278 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44317
I20260927 22:53:22.124660 12278 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:53:22.125517 12278 heartbeater.cc:511] Master 127.8.204.190:44317 requested a full tablet report, sending...
I20260927 22:53:22.127951 12089 ts_manager.cc:194] Registered new tserver with Master: 4462a787f77c4d5283b4980de4b92856 (127.8.204.129:41509)
I20260927 22:53:22.130108 12089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:38277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:22.149995 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44317
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:22.460413 12284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:22.460664 12284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:22.460754 12284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:22.467939 12284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:22.468044 12284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:22.468151 12284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:22.477227 12284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:22.478338 12284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:22.479532 12284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:22.492533 12289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:22.492568 12290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:22.492599 12292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:22.496615 12284 server_base.cc:1034] running on GCE node
I20260927 22:53:22.497222 12284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:22.498145 12284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:22.499367 12284 hybrid_clock.cc:648] HybridClock initialized: now 1790549602499305 us; error 54 us; skew 500 ppm
I20260927 22:53:22.499680 12284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:22 dist-test-slave-40vx krb5kdc[12044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:22.502771 12284 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:22.504360 12284 webserver.cc:460] Webserver started at http://127.8.204.130:43653/ using document root <none> and password file <none>
I20260927 22:53:22.505034 12284 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:22.505170 12284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:22.505471 12284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:22.508299 12284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "65aee28fa8664d77a2b155a4f8048796"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "ca02c3901bcbdb0af7424a39ec5159c9"
server_key_iv: "4855ae468c8f1eb33a68ebf268d745f1"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.509056 12284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "65aee28fa8664d77a2b155a4f8048796"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "ca02c3901bcbdb0af7424a39ec5159c9"
server_key_iv: "4855ae468c8f1eb33a68ebf268d745f1"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.513940 12284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:53:22.517268 12299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:22.518471 12284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 22:53:22.518642 12284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "65aee28fa8664d77a2b155a4f8048796"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "ca02c3901bcbdb0af7424a39ec5159c9"
server_key_iv: "4855ae468c8f1eb33a68ebf268d745f1"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.518828 12284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:22.532548 12284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:22.536645 12284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:22.536932 12284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:22.537822 12284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:22.539068 12284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:22.539165 12284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:22.539361 12284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:22.539436 12284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:53:22.572180 12284 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:46849
I20260927 22:53:22.572264 12411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:46849 every 8 connection(s)
I20260927 22:53:22.573465 12284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:22.576695 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12284
I20260927 22:53:22.576882 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:22.577422 9010 external_mini_cluster.cc:1428] Setting key e028e9ba31e1f120dd686013c67b73e3
Sep 27 22:53:22 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:53:22.600279 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:22.575952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:36595 (local address 127.8.204.190:44317)
0927 22:53:22.576130 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:22.576134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:22.577408 (+ 1274us) server_negotiation.cc:407] Connection header received
0927 22:53:22.579552 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:22.579556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:22.579618 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:22.579697 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:22.581773 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.582437 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.585162 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.585885 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.586037 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:22.591886 (+ 5849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:22.591896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:22.591905 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:22.591948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:22.595920 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:22.597489 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:22.597496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:22.597503 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:22.597655 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:22.598213 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:22.598220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:22.598226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:22.598651 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:22.598822 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:22.599705 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 22:53:22.599942 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
I20260927 22:53:22.601363 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:22.576433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44317 (local address 127.8.204.130:36595)
0927 22:53:22.577239 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:22.577287 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:22.579258 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:22.579961 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:22.579972 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:22.580759 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:22.581604 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:22.581633 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.582578 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.582586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:22.585000 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:22.585020 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:22.586116 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:22.586126 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:22.586456 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:22.588112 (+ 1656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:22.588152 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:22.591603 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:22.596106 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:22.596119 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:22.596141 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:22.596823 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:22.597772 (+ 949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:22.597778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:22.597783 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:22.598060 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:22.598903 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:22.598912 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:22.599342 (+ 430us) client_negotiation.cc:769] Sending connection context
0927 22:53:22.599613 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 22:53:22.600152 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":306,"threads_started":1}
I20260927 22:53:22.602864 12412 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44317
I20260927 22:53:22.603179 12412 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:22.603914 12412 heartbeater.cc:511] Master 127.8.204.190:44317 requested a full tablet report, sending...
I20260927 22:53:22.605316 12089 ts_manager.cc:194] Registered new tserver with Master: 65aee28fa8664d77a2b155a4f8048796 (127.8.204.130:46849)
I20260927 22:53:22.606305 12089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:36595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:22.628618 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44317
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:22.945791 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:22.946050 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:22.946236 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:22.953274 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:22.953372 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:22.953537 12417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:22.962936 12417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:22.964376 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:22.965744 12417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:22.978751 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:22.978751 12422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:22.978781 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:22.979665 12417 server_base.cc:1034] running on GCE node
I20260927 22:53:22.980201 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:22.981027 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:22.982306 12417 hybrid_clock.cc:648] HybridClock initialized: now 1790549602982243 us; error 51 us; skew 500 ppm
I20260927 22:53:22.982612 12417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:22 dist-test-slave-40vx krb5kdc[12044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:22.985697 12417 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:22.986997 12417 webserver.cc:460] Webserver started at http://127.8.204.131:39049/ using document root <none> and password file <none>
I20260927 22:53:22.987717 12417 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:22.987862 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:22.988170 12417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:22.990898 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "466dfc3b277a46d9820576835440ae4f"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "2db69cb37a5f3b445917913b0b8daedf"
server_key_iv: "ef08d8f30737d5e7abe6e34c647e656b"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.991673 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "466dfc3b277a46d9820576835440ae4f"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "2db69cb37a5f3b445917913b0b8daedf"
server_key_iv: "ef08d8f30737d5e7abe6e34c647e656b"
server_key_version: "encryptionkey@0"
I20260927 22:53:22.996500 12417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:53:22.999696 12432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:23.000885 12417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:53:23.001057 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "466dfc3b277a46d9820576835440ae4f"
format_stamp: "Formatted at 2026-09-27 22:53:22 on dist-test-slave-40vx"
server_key: "2db69cb37a5f3b445917913b0b8daedf"
server_key_iv: "ef08d8f30737d5e7abe6e34c647e656b"
server_key_version: "encryptionkey@0"
I20260927 22:53:23.001253 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:23.015751 12417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:23.020193 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:23.020471 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:23.021371 12417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:23.022783 12417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:23.022876 12417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:23.023034 12417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:23.023108 12417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:23.055500 12417 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36609
I20260927 22:53:23.055523 12544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36609 every 8 connection(s)
I20260927 22:53:23.056744 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:23.065573 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12417
I20260927 22:53:23.065754 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:23.066326 9010 external_mini_cluster.cc:1428] Setting key 079cb6995075116e733dbb1121a784f5
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549602, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:23.075938 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.059146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:42413 (local address 127.8.204.190:44317)
0927 22:53:23.059359 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:23.059364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:23.060549 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 22:53:23.062284 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:23.062290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:23.062393 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:23.062500 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:23.064018 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.064822 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.066766 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.067370 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.067499 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.070748 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:23.070754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:23.070759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:23.070800 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:23.073130 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.073794 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.073801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.073805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.073933 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.074326 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.074333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.074336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.074587 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:23.074728 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:23.075389 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 22:53:23.075589 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:53:23.076835 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.059733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44317 (local address 127.8.204.131:42413)
0927 22:53:23.060388 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:23.060427 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:23.062021 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:23.062736 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:23.062744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:23.063215 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:23.063857 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:23.063873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.064972 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.064976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:23.066617 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:23.066629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.067590 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.067600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:23.067832 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.068499 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:23.068521 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:23.070564 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:23.073267 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:23.073275 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:23.073290 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:23.073679 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:23.074036 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:23.074043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:23.074048 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:23.074219 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:23.074715 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:23.074723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:23.075035 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 22:53:23.075310 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 22:53:23.075812 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":261,"threads_started":1}
I20260927 22:53:23.078033 12545 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44317
I20260927 22:53:23.078331 12545 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:23.079007 12545 heartbeater.cc:511] Master 127.8.204.190:44317 requested a full tablet report, sending...
I20260927 22:53:23.080554 12089 ts_manager.cc:194] Registered new tserver with Master: 466dfc3b277a46d9820576835440ae4f (127.8.204.131:36609)
I20260927 22:53:23.081266 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:53:23.081521 12089 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:42413
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:23.126363 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.113412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.8.204.190:44317)
0927 22:53:23.113583 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:23.113588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:23.114025 (+ 437us) server_negotiation.cc:407] Connection header received
0927 22:53:23.114167 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:23.114171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:23.114231 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:23.114319 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:23.115633 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.116324 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.117501 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.117912 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.118017 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.121588 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:23.121593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:23.121596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:23.121649 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:23.124302 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.124849 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.124853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.124855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.124971 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.125288 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.125291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.125293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.125617 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:23.125785 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:23.125995 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:53:23.126139 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:53:23.126914 12089 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36960
I20260927 22:53:23.128554 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12150
I20260927 22:53:23.132910 12278 heartbeater.cc:503] Master 127.8.204.190:44317 was elected leader, sending a full tablet report...
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:23.143752 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.129463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.8.204.129:41509)
0927 22:53:23.130062 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:23.130069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:23.130092 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:53:23.130181 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:23.130186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:23.130389 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:23.130507 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:23.131596 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.132551 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.133747 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.134425 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.134546 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.138347 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:23.138357 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:23.138370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:23.138407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:23.140985 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.141594 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.141604 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.141607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.141749 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.142146 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.142153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.142156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.142577 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:23.142880 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:23.143148 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:53:23.143437 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":316,"threads_started":1}
W20260927 22:53:23.144929 12273 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52390
W20260927 22:53:23.145955 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:23.146167 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12150
I20260927 22:53:23.162549 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12284
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:23.177006 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.163353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.8.204.130:46849)
0927 22:53:23.163809 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:23.163815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:23.163838 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:53:23.163902 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:23.163908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:23.164100 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:23.164252 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:23.165306 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.166438 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.167678 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.168173 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.168265 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.171937 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:23.171947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:23.171960 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:23.171988 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:23.174653 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.175228 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.175234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.175236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.175400 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.175768 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.175771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.175773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.176064 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:23.176282 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:23.176549 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 22:53:23.176731 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":241,"threads_started":1}
W20260927 22:53:23.178220 12407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42312
W20260927 22:53:23.179343 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:23.179476 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12284
I20260927 22:53:23.194411 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12417
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549603, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:23.210454 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:23.195183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.8.204.131:36609)
0927 22:53:23.195804 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:23.195810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:23.195837 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:53:23.195906 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:23.195911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:23.196111 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:23.196231 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:23.197431 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.198439 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.199641 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:23.200180 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:23.200276 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:23.204853 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:23.204864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:23.204877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:23.204907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:23.207747 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.208396 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.208402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.208405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.208515 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:23.208862 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:23.208868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:23.208872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:23.209199 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:23.209500 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:23.209855 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 22:53:23.210130 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":338,"threads_started":1}
W20260927 22:53:23.211705 12540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52886
W20260927 22:53:23.212813 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12417: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:53:23.213022 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12417
I20260927 22:53:23.227818 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12057
I20260927 22:53:23.228948 12119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:23.329885 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12057
2026-09-27T22:53:23Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4475 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12562](info): set up 2 sockets
Sep 27 22:53:23 dist-test-slave-40vx krb5kdc[12562](info): commencing operation
krb5kdc: starting...
W20260927 22:53:24.411628 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:24 dist-test-slave-40vx krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:24.595109 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40123
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:24.916035 12575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:24.916317 12575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:24.916478 12575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:24.923640 12575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:24.923760 12575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:24.923846 12575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:24.923949 12575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:24.924036 12575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:24.933586 12575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:24.934828 12575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:24.936144 12575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:24.947808 12581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:24.947890 12580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:24.947902 12583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:24.948772 12575 server_base.cc:1034] running on GCE node
I20260927 22:53:24.949455 12575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:24.950738 12575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:24.951939 12575 hybrid_clock.cc:648] HybridClock initialized: now 1790549604951901 us; error 29 us; skew 500 ppm
I20260927 22:53:24.952296 12575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:24 dist-test-slave-40vx krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:24.955454 12575 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:24.956846 12575 webserver.cc:460] Webserver started at http://127.8.204.190:41405/ using document root <none> and password file <none>
I20260927 22:53:24.957510 12575 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:24.957653 12575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:24.957953 12575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:24.960865 12575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "295103e23a224e748a64ded2cfc056f3"
format_stamp: "Formatted at 2026-09-27 22:53:24 on dist-test-slave-40vx"
server_key: "fab7051790674d679521bf2a06738711"
server_key_iv: "fe244c46d414872e2f0753977d66b848"
server_key_version: "encryptionkey@0"
I20260927 22:53:24.961644 12575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "295103e23a224e748a64ded2cfc056f3"
format_stamp: "Formatted at 2026-09-27 22:53:24 on dist-test-slave-40vx"
server_key: "fab7051790674d679521bf2a06738711"
server_key_iv: "fe244c46d414872e2f0753977d66b848"
server_key_version: "encryptionkey@0"
I20260927 22:53:24.966497 12575 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:53:24.969792 12590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:24.971194 12575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:53:24.971380 12575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "295103e23a224e748a64ded2cfc056f3"
format_stamp: "Formatted at 2026-09-27 22:53:24 on dist-test-slave-40vx"
server_key: "fab7051790674d679521bf2a06738711"
server_key_iv: "fe244c46d414872e2f0753977d66b848"
server_key_version: "encryptionkey@0"
I20260927 22:53:24.971575 12575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:24.987550 12575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:24.992143 12575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:24.992440 12575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:25.011135 12575 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40123
I20260927 22:53:25.011155 12641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40123 every 8 connection(s)
I20260927 22:53:25.012542 12575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:25.016228 12642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:25.021350 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12575
I20260927 22:53:25.021569 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:25.022286 9010 external_mini_cluster.cc:1428] Setting key d09d2f3dba4d674dbf0b95002c59ad3b
I20260927 22:53:25.027521 12642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3: Bootstrap starting.
I20260927 22:53:25.031082 12642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:25.032289 12642 log.cc:824] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:25.035604 12642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3: No bootstrap required, opened a new log
Sep 27 22:53:25 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:25.046010 12642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295103e23a224e748a64ded2cfc056f3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40123 } }
I20260927 22:53:25.046463 12642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:25.046574 12642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 295103e23a224e748a64ded2cfc056f3, State: Initialized, Role: FOLLOWER
I20260927 22:53:25.046814 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:25.024392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.8.204.190:40123)
0927 22:53:25.025175 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:25.025186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:25.025228 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:53:25.026903 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:25.026925 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:25.027373 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:25.027764 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:25.029486 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.030727 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.032588 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.033680 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.033814 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:25.038359 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:25.038369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:25.038393 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:25.038433 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:25.042520 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:25.043141 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:25.043149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:25.043158 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:25.043400 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:25.043780 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:25.043785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:25.043788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:25.044488 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:25.044771 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:25.045098 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 22:53:25.045490 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":344,"threads_started":1}
I20260927 22:53:25.047222 12642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295103e23a224e748a64ded2cfc056f3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40123 } }
I20260927 22:53:25.047547 12642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:25.047685 12642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:25.047845 12642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:25.050242 12642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295103e23a224e748a64ded2cfc056f3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40123 } }
I20260927 22:53:25.050654 12642 leader_election.cc:302] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 295103e23a224e748a64ded2cfc056f3; no voters:
I20260927 22:53:25.051380 12642 leader_election.cc:288] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:25.051443 12647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:25.052588 12647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [term 1 LEADER]: Becoming Leader. State: Replica: 295103e23a224e748a64ded2cfc056f3, State: Running, Role: LEADER
I20260927 22:53:25.053063 12647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295103e23a224e748a64ded2cfc056f3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40123 } }
I20260927 22:53:25.053316 12642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:25.055641 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "295103e23a224e748a64ded2cfc056f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295103e23a224e748a64ded2cfc056f3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40123 } } }
I20260927 22:53:25.056046 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:25.055697 12649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 295103e23a224e748a64ded2cfc056f3. Latest consensus state: current_term: 1 leader_uuid: "295103e23a224e748a64ded2cfc056f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "295103e23a224e748a64ded2cfc056f3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40123 } } }
I20260927 22:53:25.056658 12649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:25.057672 12652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:25.062932 12652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:25.071866 12652 catalog_manager.cc:1347] Generated new cluster ID: b2ae75f5330b4b7a940b600bc0a2243f
I20260927 22:53:25.071985 12652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:25.089308 12652 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:25.090574 12652 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:25.104415 12652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 295103e23a224e748a64ded2cfc056f3: Generated new TSK 0
I20260927 22:53:25.105239 12652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:25.150514 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40123
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:25.476540 12668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:25.476891 12668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:25.476996 12668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:25.484157 12668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:25.484261 12668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:25.484368 12668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:25.493829 12668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:25.494951 12668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:25.496207 12668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:25.508687 12673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:25.508716 12674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:25.508891 12676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:25.509443 12668 server_base.cc:1034] running on GCE node
I20260927 22:53:25.509943 12668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:25.510749 12668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:25.511931 12668 hybrid_clock.cc:648] HybridClock initialized: now 1790549605511881 us; error 42 us; skew 500 ppm
I20260927 22:53:25.512173 12668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:25 dist-test-slave-40vx krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:25.514961 12668 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:25.516331 12668 webserver.cc:460] Webserver started at http://127.8.204.129:40193/ using document root <none> and password file <none>
I20260927 22:53:25.516984 12668 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:25.517131 12668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:25.517477 12668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:25.520210 12668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "2b3825165bd44faba749e60bbca9aaef"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-40vx"
server_key: "02e090dc24421eba2f6fa9047a3b9c6c"
server_key_iv: "b4f3a729e0ab05788f7534852012050c"
server_key_version: "encryptionkey@0"
I20260927 22:53:25.520953 12668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "2b3825165bd44faba749e60bbca9aaef"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-40vx"
server_key: "02e090dc24421eba2f6fa9047a3b9c6c"
server_key_iv: "b4f3a729e0ab05788f7534852012050c"
server_key_version: "encryptionkey@0"
I20260927 22:53:25.525610 12668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:53:25.528982 12683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:25.530170 12668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:53:25.530356 12668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "2b3825165bd44faba749e60bbca9aaef"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-40vx"
server_key: "02e090dc24421eba2f6fa9047a3b9c6c"
server_key_iv: "b4f3a729e0ab05788f7534852012050c"
server_key_version: "encryptionkey@0"
I20260927 22:53:25.530567 12668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:25.549587 12668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:25.554095 12668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:25.554433 12668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:25.555532 12668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:25.556787 12668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:25.556890 12668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:25.557041 12668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:25.557112 12668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:25.589985 12668 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:38655
I20260927 22:53:25.590029 12795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:38655 every 8 connection(s)
I20260927 22:53:25.591224 12668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:25.598867 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12668
I20260927 22:53:25.599092 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:25.599920 9010 external_mini_cluster.cc:1428] Setting key 28cabaf60e6834900545832e5011b646
Sep 27 22:53:25 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:25.611346 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:25.593682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:39653 (local address 127.8.204.190:40123)
0927 22:53:25.594088 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:25.594095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:25.595072 (+ 977us) server_negotiation.cc:407] Connection header received
0927 22:53:25.596798 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:25.596802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:25.596861 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:25.596954 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:25.598318 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.599048 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.601200 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.601832 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.601942 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:25.606007 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:25.606012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:25.606015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:25.606043 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:25.608516 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:25.609229 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:25.609238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:25.609240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:25.609349 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:25.609744 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:25.609749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:25.609752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:25.609994 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:25.610153 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:25.610759 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 22:53:25.610969 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":184,"threads_started":1}
I20260927 22:53:25.612205 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:25.594273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40123 (local address 127.8.204.129:39653)
0927 22:53:25.594916 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:25.594952 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:25.596564 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:25.597155 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:25.597163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:25.597656 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:25.598168 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:25.598184 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.599208 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.599213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:25.601071 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:25.601082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:25.602060 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:25.602067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:25.602337 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:25.603041 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:25.603081 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:25.605798 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:25.608659 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:25.608668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:25.608683 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:25.609107 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:25.609453 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:25.609458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:25.609462 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:25.609633 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:25.610112 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:25.610119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:25.610424 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 22:53:25.610663 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 22:53:25.611137 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":249,"threads_started":1}
I20260927 22:53:25.613592 12796 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40123
I20260927 22:53:25.613862 12796 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:25.614533 12796 heartbeater.cc:511] Master 127.8.204.190:40123 requested a full tablet report, sending...
I20260927 22:53:25.616452 12607 ts_manager.cc:194] Registered new tserver with Master: 2b3825165bd44faba749e60bbca9aaef (127.8.204.129:38655)
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:53:25.617846 12607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:39653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:25.641662 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40123
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:25.956727 12802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:25.957142 12802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:25.957338 12802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:25.964764 12802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:25.964877 12802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:25.965003 12802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:25.975206 12802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:25.976708 12802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:25.978003 12802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:25.990854 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:25.990811 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:25.990787 12810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:25.991770 12802 server_base.cc:1034] running on GCE node
I20260927 22:53:25.992381 12802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:25.993299 12802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:25.994511 12802 hybrid_clock.cc:648] HybridClock initialized: now 1790549605994468 us; error 30 us; skew 500 ppm
I20260927 22:53:25.994881 12802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:25 dist-test-slave-40vx krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:25.998072 12802 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:25.999555 12802 webserver.cc:460] Webserver started at http://127.8.204.130:37013/ using document root <none> and password file <none>
I20260927 22:53:26.000269 12802 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:26.000442 12802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:26.000764 12802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:26.003880 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "d76e3b7c49e0434b8a8c047fd38063af"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-40vx"
server_key: "9936a6d5f73c94a03fac0a3e9db449e0"
server_key_iv: "61487c173c8100b224d23e9b73dcb744"
server_key_version: "encryptionkey@0"
I20260927 22:53:26.004769 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "d76e3b7c49e0434b8a8c047fd38063af"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-40vx"
server_key: "9936a6d5f73c94a03fac0a3e9db449e0"
server_key_iv: "61487c173c8100b224d23e9b73dcb744"
server_key_version: "encryptionkey@0"
I20260927 22:53:26.010063 12802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:53:26.013841 12817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.015376 12802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:53:26.015556 12802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "d76e3b7c49e0434b8a8c047fd38063af"
format_stamp: "Formatted at 2026-09-27 22:53:25 on dist-test-slave-40vx"
server_key: "9936a6d5f73c94a03fac0a3e9db449e0"
server_key_iv: "61487c173c8100b224d23e9b73dcb744"
server_key_version: "encryptionkey@0"
I20260927 22:53:26.015765 12802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:26.030936 12802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:26.034803 12802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:26.035064 12802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:26.036002 12802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:26.037218 12802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:26.037318 12802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.037458 12802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:26.037535 12802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.070035 12802 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:42715
I20260927 22:53:26.070127 12929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:42715 every 8 connection(s)
I20260927 22:53:26.071316 12802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:26.078876 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12802
I20260927 22:53:26.079072 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:26.079525 9010 external_mini_cluster.cc:1428] Setting key b31c8cffdd16be8a15862014b79e63ca
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549605, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:26.092051 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.073825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:51677 (local address 127.8.204.190:40123)
0927 22:53:26.073990 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.073994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.075215 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 22:53:26.077076 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.077081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.077166 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.077272 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.078748 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.079717 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.081640 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.082011 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.082105 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.086572 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.086576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.086580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.086619 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.089161 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.089853 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.089860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.089863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.089990 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.090405 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.090412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.090415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.090657 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.090816 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.091497 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.091700 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:53:26.092990 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.074334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40123 (local address 127.8.204.130:51677)
0927 22:53:26.075049 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.075091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.076811 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.077492 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.077500 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.077996 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:26.078565 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.078590 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.079880 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.079885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.081462 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.081473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.082231 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.082240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.082591 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.083364 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:26.083398 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:26.086317 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:26.089299 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:26.089308 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:26.089325 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:26.089737 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:26.090109 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:26.090114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:26.090118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:26.090299 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:26.090781 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:26.090788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:26.091115 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.091427 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.091940 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260927 22:53:26.094447 12930 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40123
I20260927 22:53:26.094745 12930 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:26.095496 12930 heartbeater.cc:511] Master 127.8.204.190:40123 requested a full tablet report, sending...
I20260927 22:53:26.096838 12607 ts_manager.cc:194] Registered new tserver with Master: d76e3b7c49e0434b8a8c047fd38063af (127.8.204.130:42715)
I20260927 22:53:26.097608 12607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:51677
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:26.122891 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40123
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:44379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:26.432418 12935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:26.432875 12935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:26.433032 12935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:26.440286 12935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:26.440431 12935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:26.440613 12935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:26.449954 12935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:44379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:26.451167 12935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:26.452437 12935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:26.465547 12941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:26.465584 12940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:26.465669 12943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:26.466120 12935 server_base.cc:1034] running on GCE node
I20260927 22:53:26.466706 12935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:26.467701 12935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:26.468916 12935 hybrid_clock.cc:648] HybridClock initialized: now 1790549606468865 us; error 46 us; skew 500 ppm
I20260927 22:53:26.469225 12935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:26.475683 12935 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:26.477092 12935 webserver.cc:460] Webserver started at http://127.8.204.131:33577/ using document root <none> and password file <none>
I20260927 22:53:26.477756 12935 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:26.477903 12935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:26.478197 12935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:26.481097 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "f5b9eec751af4e26936ed5fd9cfc4dc3"
format_stamp: "Formatted at 2026-09-27 22:53:26 on dist-test-slave-40vx"
server_key: "d48deb287a1f48cbc99a737d108a3635"
server_key_iv: "a2d67077a9ac7168e679662cd2e06436"
server_key_version: "encryptionkey@0"
I20260927 22:53:26.481865 12935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "f5b9eec751af4e26936ed5fd9cfc4dc3"
format_stamp: "Formatted at 2026-09-27 22:53:26 on dist-test-slave-40vx"
server_key: "d48deb287a1f48cbc99a737d108a3635"
server_key_iv: "a2d67077a9ac7168e679662cd2e06436"
server_key_version: "encryptionkey@0"
I20260927 22:53:26.486977 12935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:53:26.490494 12950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.491866 12935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:26.492035 12935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "f5b9eec751af4e26936ed5fd9cfc4dc3"
format_stamp: "Formatted at 2026-09-27 22:53:26 on dist-test-slave-40vx"
server_key: "d48deb287a1f48cbc99a737d108a3635"
server_key_iv: "a2d67077a9ac7168e679662cd2e06436"
server_key_version: "encryptionkey@0"
I20260927 22:53:26.492235 12935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:26.511983 12935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:26.515843 12935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:26.516120 12935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:26.517064 12935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:26.518344 12935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:26.518442 12935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.518615 12935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:26.518692 12935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:26.550624 12935 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:39463
I20260927 22:53:26.550658 13062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:39463 every 8 connection(s)
I20260927 22:53:26.551890 12935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:26.559556 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 12935
I20260927 22:53:26.559775 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:26.560292 9010 external_mini_cluster.cc:1428] Setting key fea7c102503562e1e3b059573aa01c1f
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:26.571171 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.554325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:56409 (local address 127.8.204.190:40123)
0927 22:53:26.554516 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.554522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.555770 (+ 1248us) server_negotiation.cc:407] Connection header received
0927 22:53:26.557476 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.557480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.557550 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.557637 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.559027 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.559977 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.562283 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.562901 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.563024 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.566267 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.566271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.566275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.566302 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.568610 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.569230 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.569233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.569235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.569349 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.569697 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.569700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.569702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.569936 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.570054 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.570768 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.570923 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:53:26.572180 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.554878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40123 (local address 127.8.204.131:56409)
0927 22:53:26.555627 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:26.555663 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:26.557219 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:26.557848 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:26.557856 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:26.558363 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:26.558892 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.558907 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.560128 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.560133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.562146 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:26.562158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.563148 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.563156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:26.563442 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.564109 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:26.564134 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:26.566103 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:26.568726 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:26.568732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:26.568743 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:26.569129 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:26.569440 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:26.569443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:26.569445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:26.569610 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:26.570072 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:26.570078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:26.570448 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 22:53:26.570701 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 22:53:26.571072 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":323,"threads_started":1}
I20260927 22:53:26.573518 13063 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40123
I20260927 22:53:26.573848 13063 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:26.574730 13063 heartbeater.cc:511] Master 127.8.204.190:40123 requested a full tablet report, sending...
I20260927 22:53:26.576179 12607 ts_manager.cc:194] Registered new tserver with Master: f5b9eec751af4e26936ed5fd9cfc4dc3 (127.8.204.131:39463)
I20260927 22:53:26.576932 12607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:56409
I20260927 22:53:26.587232 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260927 22:53:26.620811 12796 heartbeater.cc:503] Master 127.8.204.190:40123 was elected leader, sending a full tablet report...
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:26.633884 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.619673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.8.204.129:38655)
0927 22:53:26.620339 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.620347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.620372 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:53:26.620483 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.620489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.620717 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.620853 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.622462 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.623530 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.624659 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.625173 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.625263 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.628770 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.628779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.628793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.628821 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.631653 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.632193 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.632200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.632202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.632312 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.632594 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.632598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.632600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.632896 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.633113 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.633293 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.633465 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":358,"threads_started":1}
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:26.650951 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.638131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58910 (local address 127.8.204.190:40123)
0927 22:53:26.638360 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.638364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.638453 (+ 89us) server_negotiation.cc:407] Connection header received
0927 22:53:26.638585 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.638589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.638648 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.638727 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.639880 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.640587 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.642286 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.642677 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.642772 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.646288 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.646293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.646296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.646327 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.648844 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.649387 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.649390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.649392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.649510 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.649784 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.649786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.649789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.650085 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.650251 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.650475 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.650622 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 22:53:26.651643 12607 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58910
I20260927 22:53:26.652611 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12668
I20260927 22:53:26.665241 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.653441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.8.204.129:38655)
0927 22:53:26.653662 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.653668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.653880 (+ 212us) server_negotiation.cc:407] Connection header received
0927 22:53:26.654074 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.654080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.654178 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.654282 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.655396 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.656067 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.657225 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.657750 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.657886 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.660356 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.660362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.660368 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.660407 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.662767 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.663412 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.663419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.663423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.663547 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.663911 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.663916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.663920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.664172 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.664411 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.664709 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.664923 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:53:26.665920 12791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:26.749425 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12668
I20260927 22:53:26.765563 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12802
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:26.780308 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.766452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.8.204.130:42715)
0927 22:53:26.766959 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.766967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.766989 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:53:26.767050 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.767056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.767261 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.767445 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.768522 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.769475 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.770644 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.771170 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.771260 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.775329 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.775341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.775355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.775385 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.777981 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.778560 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.778567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.778569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.778676 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.778969 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.778972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.778974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.779346 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.779562 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.779837 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.780018 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260927 22:53:26.781536 12925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:26.862704 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12802
I20260927 22:53:26.878238 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12935
Sep 27 22:53:26 dist-test-slave-40vx krb5kdc[12562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:26.893888 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:26.879169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.8.204.131:39463)
0927 22:53:26.879816 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:26.879823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:26.879851 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:53:26.879912 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:26.879919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:26.880137 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:26.880318 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:26.881595 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.882617 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.883974 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:26.884528 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:26.884670 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:26.888951 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:26.888962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:26.888977 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:26.889013 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:26.891592 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.892138 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.892144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.892147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.892259 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:26.892557 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:26.892560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:26.892562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:26.892867 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:26.893120 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:26.893314 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:53:26.893497 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":354,"threads_started":1}
I20260927 22:53:26.895227 13058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:26.978562 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12935
I20260927 22:53:26.994249 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 12575
I20260927 22:53:26.995464 12637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:27.096709 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 12575
2026-09-27T22:53:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3763 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:27 dist-test-slave-40vx krb5kdc[13082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:27 dist-test-slave-40vx krb5kdc[13082](info): set up 2 sockets
Sep 27 22:53:27 dist-test-slave-40vx krb5kdc[13082](info): commencing operation
krb5kdc: starting...
W20260927 22:53:28.163793 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:28 dist-test-slave-40vx krb5kdc[13082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:28.349329 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40787
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:28.661118 13095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:28.661383 13095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:28.661485 13095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:28.668617 13095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:28.668718 13095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:28.668763 13095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:28.668813 13095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:28.668852 13095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:28.678351 13095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:28.679687 13095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:28.680965 13095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:28.692256 13100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:28.692227 13103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:28.692291 13101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:28.693188 13095 server_base.cc:1034] running on GCE node
I20260927 22:53:28.693961 13095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:28.695382 13095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:28.696621 13095 hybrid_clock.cc:648] HybridClock initialized: now 1790549608696564 us; error 56 us; skew 500 ppm
I20260927 22:53:28.696965 13095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:28 dist-test-slave-40vx krb5kdc[13082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549608, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:28.699934 13095 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:28.701128 13095 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:28.701260 13095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:28.701575 13095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:28.704471 13095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "6330dca416c347d0b348589595f46049"
format_stamp: "Formatted at 2026-09-27 22:53:28 on dist-test-slave-40vx"
server_key: "474a03df693e28cdad638e08dbb079df"
server_key_iv: "f9dc994dca1d91a925e22972c1aa757f"
server_key_version: "encryptionkey@0"
I20260927 22:53:28.705237 13095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "6330dca416c347d0b348589595f46049"
format_stamp: "Formatted at 2026-09-27 22:53:28 on dist-test-slave-40vx"
server_key: "474a03df693e28cdad638e08dbb079df"
server_key_iv: "f9dc994dca1d91a925e22972c1aa757f"
server_key_version: "encryptionkey@0"
I20260927 22:53:28.709920 13095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:53:28.713140 13109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:28.714376 13095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:28.714555 13095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "6330dca416c347d0b348589595f46049"
format_stamp: "Formatted at 2026-09-27 22:53:28 on dist-test-slave-40vx"
server_key: "474a03df693e28cdad638e08dbb079df"
server_key_iv: "f9dc994dca1d91a925e22972c1aa757f"
server_key_version: "encryptionkey@0"
I20260927 22:53:28.714762 13095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:28.745440 13095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:28.749287 13095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:28.749558 13095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:28.768159 13095 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40787
I20260927 22:53:28.768167 13160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40787 every 8 connection(s)
I20260927 22:53:28.768993 13095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:28.772400 13161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:28.777005 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13095
I20260927 22:53:28.777233 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:28.777858 9010 external_mini_cluster.cc:1428] Setting key 6d6029f5431402e78749a422f19a53f5
I20260927 22:53:28.784370 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049: Bootstrap starting.
I20260927 22:53:28.787807 13161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:28.788852 13161 log.cc:824] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:28.791437 13161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049: No bootstrap required, opened a new log
Sep 27 22:53:28 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:28.801476 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:28.779731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.8.204.190:40787)
0927 22:53:28.780587 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:28.780601 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:28.780651 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:53:28.782954 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:28.782978 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:28.783443 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:28.783837 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:28.785170 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:28.787174 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:28.788463 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:28.789523 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:28.789659 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:28.793568 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:28.793581 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:28.793603 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:28.793648 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:28.797410 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:28.798002 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:28.798010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:28.798017 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:28.798244 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:28.798549 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:28.798554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:28.798557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:28.799233 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:28.799533 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:28.799880 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:53:28.800275 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":355,"threads_started":1}
I20260927 22:53:28.801252 13161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330dca416c347d0b348589595f46049" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40787 } }
I20260927 22:53:28.801635 13161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:28.801762 13161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6330dca416c347d0b348589595f46049, State: Initialized, Role: FOLLOWER
I20260927 22:53:28.802294 13161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330dca416c347d0b348589595f46049" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40787 } }
I20260927 22:53:28.802505 13161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:28.802634 13161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:28.802800 13161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:28.805003 13161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330dca416c347d0b348589595f46049" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40787 } }
I20260927 22:53:28.805444 13161 leader_election.cc:302] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6330dca416c347d0b348589595f46049; no voters:
I20260927 22:53:28.805992 13161 leader_election.cc:288] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:28.806116 13166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:28.806798 13166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [term 1 LEADER]: Becoming Leader. State: Replica: 6330dca416c347d0b348589595f46049, State: Running, Role: LEADER
I20260927 22:53:28.807335 13166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330dca416c347d0b348589595f46049" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40787 } }
I20260927 22:53:28.808008 13161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:28.809913 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6330dca416c347d0b348589595f46049. Latest consensus state: current_term: 1 leader_uuid: "6330dca416c347d0b348589595f46049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330dca416c347d0b348589595f46049" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40787 } } }
I20260927 22:53:28.810178 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6330dca416c347d0b348589595f46049" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330dca416c347d0b348589595f46049" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40787 } } }
I20260927 22:53:28.810423 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:28.810420 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:28.811399 13174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:28.817555 13174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:28.827105 13174 catalog_manager.cc:1347] Generated new cluster ID: 25698d10385a4bb19b00a4bd4041850c
I20260927 22:53:28.827220 13174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:28.838423 13174 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:28.840276 13174 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:28.858475 13174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049: Generated new TSK 0
I20260927 22:53:28.859485 13174 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:28.910879 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40787
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:29.227246 13187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:29.227579 13187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:29.227756 13187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:29.234771 13187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:29.234866 13187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:29.235051 13187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:29.244930 13187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:29.246235 13187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:29.247644 13187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:29.260685 13192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:29.260900 13195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:29.260725 13193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:29.261718 13187 server_base.cc:1034] running on GCE node
I20260927 22:53:29.262322 13187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:29.263255 13187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:29.264546 13187 hybrid_clock.cc:648] HybridClock initialized: now 1790549609264487 us; error 55 us; skew 500 ppm
I20260927 22:53:29.264866 13187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:29 dist-test-slave-40vx krb5kdc[13082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:29.267953 13187 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:29.269109 13187 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:29.269239 13187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:29.269549 13187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:29.272392 13187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "3fb151e4e73443d3a386c8958016b67f"
format_stamp: "Formatted at 2026-09-27 22:53:29 on dist-test-slave-40vx"
server_key: "ed2978c7794360d14901cc62c4c59833"
server_key_iv: "7461fdc2b52a688e16b754b2e0bdce6d"
server_key_version: "encryptionkey@0"
I20260927 22:53:29.273188 13187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "3fb151e4e73443d3a386c8958016b67f"
format_stamp: "Formatted at 2026-09-27 22:53:29 on dist-test-slave-40vx"
server_key: "ed2978c7794360d14901cc62c4c59833"
server_key_iv: "7461fdc2b52a688e16b754b2e0bdce6d"
server_key_version: "encryptionkey@0"
I20260927 22:53:29.278194 13187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 22:53:29.281826 13201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:29.283208 13187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:53:29.283413 13187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "3fb151e4e73443d3a386c8958016b67f"
format_stamp: "Formatted at 2026-09-27 22:53:29 on dist-test-slave-40vx"
server_key: "ed2978c7794360d14901cc62c4c59833"
server_key_iv: "7461fdc2b52a688e16b754b2e0bdce6d"
server_key_version: "encryptionkey@0"
I20260927 22:53:29.283627 13187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:29.302459 13187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:29.306396 13187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:29.306679 13187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:29.307765 13187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:29.309249 13187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:29.309383 13187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:29.309513 13187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:29.309607 13187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:29.342070 13187 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:42369
I20260927 22:53:29.342123 13313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:42369 every 8 connection(s)
I20260927 22:53:29.342907 13187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:29.348418 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13187
I20260927 22:53:29.348632 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:29.349264 9010 external_mini_cluster.cc:1428] Setting key c70352ed53694afb632be648eeefb219
Sep 27 22:53:29 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:29.364234 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:29.345620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:59761 (local address 127.8.204.190:40787)
0927 22:53:29.346078 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:29.346083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:29.346872 (+ 789us) server_negotiation.cc:407] Connection header received
0927 22:53:29.348762 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:29.348767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:29.348851 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:29.348978 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:29.350588 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.351577 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.353080 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.353461 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.353598 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:29.357271 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:29.357280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:29.357289 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:29.357332 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:29.361009 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:29.361737 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:29.361745 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:29.361752 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:29.361897 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:29.362279 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:29.362287 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:29.362293 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:29.362699 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:29.362856 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:29.363534 (+ 678us) server_negotiation.cc:299] Negotiation successful
0927 22:53:29.363783 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":248,"threads_started":1}
I20260927 22:53:29.365048 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:29.346052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40787 (local address 127.8.204.129:59761)
0927 22:53:29.346718 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:29.346758 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:29.348446 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:29.349208 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:29.349216 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:29.349775 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:29.350417 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:29.350434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.351728 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.351732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:29.352931 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:29.352942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.353681 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.353690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:29.353918 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:29.354570 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:29.354591 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:29.357052 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:29.361141 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:29.361147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:29.361158 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:29.361613 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:29.362002 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:29.362005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:29.362007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:29.362173 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:29.362824 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:29.362829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:29.363116 (+ 287us) client_negotiation.cc:769] Sending connection context
0927 22:53:29.363413 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 22:53:29.363926 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
I20260927 22:53:29.366642 13314 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40787
I20260927 22:53:29.366967 13314 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:29.367815 13314 heartbeater.cc:511] Master 127.8.204.190:40787 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:53:29.369912 13126 ts_manager.cc:194] Registered new tserver with Master: 3fb151e4e73443d3a386c8958016b67f (127.8.204.129:42369)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:53:29.371479 13126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:59761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:29.394470 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40787
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:29.706772 13320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:29.707042 13320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:29.707208 13320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:29.714756 13320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:29.714857 13320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:29.715073 13320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:29.726126 13320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:29.727705 13320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:29.728974 13320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:29.741508 13325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:29.741521 13328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:29.741457 13326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:29.742282 13320 server_base.cc:1034] running on GCE node
I20260927 22:53:29.742870 13320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:29.743798 13320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:29.745019 13320 hybrid_clock.cc:648] HybridClock initialized: now 1790549609744968 us; error 46 us; skew 500 ppm
I20260927 22:53:29.745352 13320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:29 dist-test-slave-40vx krb5kdc[13082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:29.748396 13320 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:29.749434 13320 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:29.749560 13320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:29.749858 13320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:29.752662 13320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "80adbd3bf7fe4c26a930c29c470dda5c"
format_stamp: "Formatted at 2026-09-27 22:53:29 on dist-test-slave-40vx"
server_key: "ba31ee43149174f8ad08c237aceded9a"
server_key_iv: "52c27089d2f1ce23369e69a93e9ad65e"
server_key_version: "encryptionkey@0"
I20260927 22:53:29.753430 13320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "80adbd3bf7fe4c26a930c29c470dda5c"
format_stamp: "Formatted at 2026-09-27 22:53:29 on dist-test-slave-40vx"
server_key: "ba31ee43149174f8ad08c237aceded9a"
server_key_iv: "52c27089d2f1ce23369e69a93e9ad65e"
server_key_version: "encryptionkey@0"
I20260927 22:53:29.758056 13320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 22:53:29.761204 13334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:29.762377 13320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:29.762539 13320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "80adbd3bf7fe4c26a930c29c470dda5c"
format_stamp: "Formatted at 2026-09-27 22:53:29 on dist-test-slave-40vx"
server_key: "ba31ee43149174f8ad08c237aceded9a"
server_key_iv: "52c27089d2f1ce23369e69a93e9ad65e"
server_key_version: "encryptionkey@0"
I20260927 22:53:29.762746 13320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:29.789938 13320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:29.793826 13320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:29.794103 13320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:29.795025 13320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:29.796370 13320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:29.796501 13320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:29.796622 13320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:29.796707 13320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:29.833904 13320 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:39271
I20260927 22:53:29.833988 13446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:39271 every 8 connection(s)
I20260927 22:53:29.834776 13320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:29.841821 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13320
I20260927 22:53:29.842029 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:29.842448 9010 external_mini_cluster.cc:1428] Setting key 901bc4693ebb5ed28722e81d86c7c7b0
Sep 27 22:53:29 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549609, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:29.853919 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:29.837172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:54355 (local address 127.8.204.190:40787)
0927 22:53:29.837337 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:29.837342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:29.838531 (+ 1189us) server_negotiation.cc:407] Connection header received
0927 22:53:29.840368 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:29.840373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:29.840465 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:29.840579 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:29.842018 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.842832 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.844402 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.844792 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.844923 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:29.848252 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:29.848258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:29.848261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:29.848291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:29.850821 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:29.851751 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:29.851755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:29.851758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:29.851875 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:29.852333 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:29.852336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:29.852338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:29.852571 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:29.852678 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:29.853502 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 22:53:29.853680 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:53:29.855468 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:29.837686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40787 (local address 127.8.204.130:54355)
0927 22:53:29.838350 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:29.838387 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:29.840108 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:29.840799 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:29.840807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:29.841309 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:29.841858 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:29.841874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.842969 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.842979 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:29.844256 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:29.844267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:29.845005 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:29.845020 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:29.845225 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:29.845915 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:29.845936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:29.848047 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:29.850956 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:29.850968 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:29.850987 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:29.851637 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:29.851984 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:29.851992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:29.851998 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:29.852227 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:29.852686 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:29.852697 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:29.853132 (+ 435us) client_negotiation.cc:769] Sending connection context
0927 22:53:29.853432 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 22:53:29.854025 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":274,"threads_started":1}
I20260927 22:53:29.856936 13447 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40787
I20260927 22:53:29.857259 13447 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:29.858069 13447 heartbeater.cc:511] Master 127.8.204.190:40787 requested a full tablet report, sending...
I20260927 22:53:29.859670 13126 ts_manager.cc:194] Registered new tserver with Master: 80adbd3bf7fe4c26a930c29c470dda5c (127.8.204.130:39271)
I20260927 22:53:29.860432 13126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:54355
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:29.886526 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40787
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:30.195833 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:30.196089 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:30.196241 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:30.202930 13452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:30.203019 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:30.203214 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:30.212774 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40787
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:30.213972 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:30.215173 13452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:30.227591 13458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:30.227613 13460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:30.227795 13457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:30.228325 13452 server_base.cc:1034] running on GCE node
I20260927 22:53:30.228904 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:30.229758 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:30.230981 13452 hybrid_clock.cc:648] HybridClock initialized: now 1790549610230939 us; error 49 us; skew 500 ppm
I20260927 22:53:30.231313 13452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:30 dist-test-slave-40vx krb5kdc[13082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549610, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:30.234294 13452 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:30.235354 13452 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:30.235440 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:30.235677 13452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:30.238371 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "f67b012deb574287b7fd0c2eb57f4671"
format_stamp: "Formatted at 2026-09-27 22:53:30 on dist-test-slave-40vx"
server_key: "f019bb3dcf50375475fff8ff90872a72"
server_key_iv: "ad34bc283f711c7c60653a705a0d6b61"
server_key_version: "encryptionkey@0"
I20260927 22:53:30.239084 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "f67b012deb574287b7fd0c2eb57f4671"
format_stamp: "Formatted at 2026-09-27 22:53:30 on dist-test-slave-40vx"
server_key: "f019bb3dcf50375475fff8ff90872a72"
server_key_iv: "ad34bc283f711c7c60653a705a0d6b61"
server_key_version: "encryptionkey@0"
I20260927 22:53:30.243707 13452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 22:53:30.246824 13466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:30.248164 13452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:53:30.248278 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "f67b012deb574287b7fd0c2eb57f4671"
format_stamp: "Formatted at 2026-09-27 22:53:30 on dist-test-slave-40vx"
server_key: "f019bb3dcf50375475fff8ff90872a72"
server_key_iv: "ad34bc283f711c7c60653a705a0d6b61"
server_key_version: "encryptionkey@0"
I20260927 22:53:30.248445 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:30.265944 13452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:30.269855 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:30.270077 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:30.270946 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:30.272277 13452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:30.272362 13452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:30.272435 13452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:30.272491 13452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:30.304869 13452 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:45517
I20260927 22:53:30.304962 13578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:45517 every 8 connection(s)
I20260927 22:53:30.305692 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:30.313232 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13452
I20260927 22:53:30.313422 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:30.313834 9010 external_mini_cluster.cc:1428] Setting key da339117e57a1d7e5fd5d2d5baad0058
Sep 27 22:53:30 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549610, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:30.324470 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.307972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:60397 (local address 127.8.204.190:40787)
0927 22:53:30.308188 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.308193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.309334 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 22:53:30.311109 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.311114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.311194 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.311317 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:30.312769 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.313631 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.315190 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.315672 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.315802 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.319012 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:30.319018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:30.319023 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:30.319062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:30.321677 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.322347 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.322354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.322357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.322477 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.322869 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.322875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.322879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.323140 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:30.323323 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.323922 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.324122 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:30.325326 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.308503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40787 (local address 127.8.204.131:60397)
0927 22:53:30.309157 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:30.309199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:30.310820 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:30.311533 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:30.311541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:30.312086 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:30.312615 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.312631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.313766 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.313786 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.315055 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.315065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.315880 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.315890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.316124 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.316785 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:30.316806 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:30.318814 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:30.321808 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:30.321817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:30.321831 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:30.322231 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:30.322586 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:30.322591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:30.322595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:30.322763 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:30.323283 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:30.323291 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:30.323600 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 22:53:30.323848 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 22:53:30.324329 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":264,"threads_started":1}
I20260927 22:53:30.326586 13579 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40787
I20260927 22:53:30.326879 13579 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:30.327617 13579 heartbeater.cc:511] Master 127.8.204.190:40787 requested a full tablet report, sending...
I20260927 22:53:30.329020 13126 ts_manager.cc:194] Registered new tserver with Master: f67b012deb574287b7fd0c2eb57f4671 (127.8.204.131:45517)
I20260927 22:53:30.329779 13126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:60397
I20260927 22:53:30.340601 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:53:30.357882 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.345735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.8.204.190:40787)
0927 22:53:30.345935 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.345942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.346418 (+ 476us) server_negotiation.cc:407] Connection header received
0927 22:53:30.346595 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.346599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.346654 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.346724 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:30.348506 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.349207 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.350410 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.350819 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.350991 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.352998 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:30.353002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:30.353006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:30.353033 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:30.355553 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.356101 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.356109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.356115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.356258 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.356678 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.356681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.356683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.356954 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:30.357123 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.357419 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.357564 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:53:30.361459 13126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:30.368320 13126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:30.374313 13314 heartbeater.cc:503] Master 127.8.204.190:40787 was elected leader, sending a full tablet report...
I20260927 22:53:30.391985 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.380131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58992 (local address 127.8.204.129:42369)
0927 22:53:30.380784 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.380792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.381384 (+ 592us) server_negotiation.cc:407] Connection header received
0927 22:53:30.381674 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.381682 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.381942 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.382135 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:30.383304 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.385002 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.388221 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.390163 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.390304 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.390586 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.391440 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.391614 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":347,"threads_started":1}
I20260927 22:53:30.393066 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.380060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42369 (local address 127.0.0.1:58992)
0927 22:53:30.381284 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:30.381300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:30.381494 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:30.382514 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:30.382524 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:30.382565 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:30.383053 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.383068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.385225 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.385232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.388059 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.388071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.390811 (+ 2740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.390820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.391124 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.391193 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:53:30.392232 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0927 22:53:30.392496 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"mutex_wait_us":621,"thread_start_us":330,"threads_started":1}
I20260927 22:53:30.400974 13249 tablet_service.cc:1467] Processing CreateTablet for tablet 36a5c5b7abaf49378e50afd8b145e646 (DEFAULT_TABLE table=test-table [id=1dacecf1552e4a07b55d26850976b377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:30.401983 13594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.381474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:45517 (local address 127.0.0.1:34954)
0927 22:53:30.393331 (+ 11857us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:30.393354 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:30.393479 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:30.396530 (+ 3051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:30.396535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:30.396560 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:30.396802 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.396813 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.398344 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.398352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.399762 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.399773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.401177 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.401182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.401363 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.401397 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:53:30.401491 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:53:30.401625 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11735,"mutex_wait_us":63,"thread_start_us":15296,"threads_started":1}
I20260927 22:53:30.402375 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.381760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.8.204.131:45517)
0927 22:53:30.395990 (+ 14230us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.395996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.396021 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:53:30.396096 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.396102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.396337 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.396547 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:30.397032 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.398195 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.399908 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.401007 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.401138 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.401353 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.401551 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.401804 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14058,"thread_start_us":13918,"threads_started":1}
I20260927 22:53:30.403146 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.380045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:39271 (local address 127.0.0.1:55846)
0927 22:53:30.380661 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:30.380698 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:30.380890 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:30.396264 (+ 15374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:30.396268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:30.396292 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:30.396696 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.396710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.398413 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.398418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.400490 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.400504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.402382 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.402389 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.402563 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.402597 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:53:30.402688 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:53:30.402826 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":619,"threads_started":1}
I20260927 22:53:30.403308 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.380583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.8.204.130:39271)
0927 22:53:30.395724 (+ 15141us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.395739 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.395764 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:53:30.395847 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.395853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.396047 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.396273 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:30.396896 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.398296 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.400658 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.402117 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.402235 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.402415 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.402844 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.403007 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15005,"thread_start_us":14919,"threads_started":1}
I20260927 22:53:30.403561 13249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a5c5b7abaf49378e50afd8b145e646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:30.406457 13382 tablet_service.cc:1467] Processing CreateTablet for tablet 36a5c5b7abaf49378e50afd8b145e646 (DEFAULT_TABLE table=test-table [id=1dacecf1552e4a07b55d26850976b377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:30.406450 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 36a5c5b7abaf49378e50afd8b145e646 (DEFAULT_TABLE table=test-table [id=1dacecf1552e4a07b55d26850976b377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:30.407856 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a5c5b7abaf49378e50afd8b145e646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:30.407856 13382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a5c5b7abaf49378e50afd8b145e646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:30.415109 13597 tablet_bootstrap.cc:492] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Bootstrap starting.
I20260927 22:53:30.418501 13597 tablet_bootstrap.cc:654] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:30.419737 13597 log.cc:824] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:30.423422 13599 tablet_bootstrap.cc:492] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Bootstrap starting.
I20260927 22:53:30.427992 13600 tablet_bootstrap.cc:492] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Bootstrap starting.
I20260927 22:53:30.428714 13599 tablet_bootstrap.cc:654] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:30.430051 13599 log.cc:824] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:30.431242 13600 tablet_bootstrap.cc:654] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:30.431442 13597 tablet_bootstrap.cc:492] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: No bootstrap required, opened a new log
I20260927 22:53:30.431725 13597 ts_tablet_manager.cc:1397] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.003s
I20260927 22:53:30.432307 13600 log.cc:824] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:30.433099 13599 tablet_bootstrap.cc:492] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: No bootstrap required, opened a new log
I20260927 22:53:30.433364 13599 ts_tablet_manager.cc:1397] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:53:30.434868 13600 tablet_bootstrap.cc:492] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: No bootstrap required, opened a new log
I20260927 22:53:30.435115 13600 ts_tablet_manager.cc:1397] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:53:30.441797 13600 raft_consensus.cc:348] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.442220 13600 raft_consensus.cc:374] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:30.442312 13600 raft_consensus.cc:729] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67b012deb574287b7fd0c2eb57f4671, State: Initialized, Role: FOLLOWER
I20260927 22:53:30.442342 13597 raft_consensus.cc:348] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.442735 13597 raft_consensus.cc:374] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:30.442817 13597 raft_consensus.cc:729] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb151e4e73443d3a386c8958016b67f, State: Initialized, Role: FOLLOWER
I20260927 22:53:30.443058 13600 consensus_queue.cc:260] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.443562 13597 consensus_queue.cc:260] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.443574 13599 raft_consensus.cc:348] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.443953 13599 raft_consensus.cc:374] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:30.444092 13599 raft_consensus.cc:729] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80adbd3bf7fe4c26a930c29c470dda5c, State: Initialized, Role: FOLLOWER
I20260927 22:53:30.444403 13579 heartbeater.cc:503] Master 127.8.204.190:40787 was elected leader, sending a full tablet report...
I20260927 22:53:30.444904 13600 ts_tablet_manager.cc:1428] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 22:53:30.444741 13599 consensus_queue.cc:260] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.446602 13597 ts_tablet_manager.cc:1428] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260927 22:53:30.447746 13447 heartbeater.cc:503] Master 127.8.204.190:40787 was elected leader, sending a full tablet report...
I20260927 22:53:30.448269 13599 ts_tablet_manager.cc:1428] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 22:53:30.529446 13603 raft_consensus.cc:482] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:30.529709 13603 raft_consensus.cc:504] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.531576 13603 leader_election.cc:288] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb151e4e73443d3a386c8958016b67f (127.8.204.129:42369), 80adbd3bf7fe4c26a930c29c470dda5c (127.8.204.130:39271)
I20260927 22:53:30.538107 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.531972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42369 (local address 127.8.204.131:34477)
0927 22:53:30.532296 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:30.532315 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:30.532434 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:30.532792 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:30.532797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:30.532824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:30.533069 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.533083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.534296 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.534300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.536309 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.536329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.537570 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.537575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.537717 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.537741 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:53:30.537818 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 22:53:30.537902 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":48}
I20260927 22:53:30.538215 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.532145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:34477 (local address 127.8.204.129:42369)
0927 22:53:30.532506 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.532513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.532539 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:53:30.532607 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.532610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.532665 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.532775 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:30.533187 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.534171 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.536462 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.537423 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.537591 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.537665 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.537856 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.537978 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260927 22:53:30.539100 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.532275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:48511 (local address 127.8.204.130:39271)
0927 22:53:30.532487 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.532493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.532948 (+ 455us) server_negotiation.cc:407] Connection header received
0927 22:53:30.533160 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.533166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.533241 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.533345 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:30.533821 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.534665 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.536920 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.538113 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.538278 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.538354 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.538747 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.538866 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:53:30.539234 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.532106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:39271 (local address 127.8.204.131:48511)
0927 22:53:30.532861 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:30.532881 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:30.533009 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:30.533361 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:30.533364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:30.533425 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:30.533686 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.533692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.534816 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.534823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.536746 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:30.536756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.538476 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.538487 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:30.538601 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.538622 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:53:30.538724 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:53:30.538841 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":3,"thread_start_us":766,"threads_started":1}
I20260927 22:53:30.539081 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a5c5b7abaf49378e50afd8b145e646" candidate_uuid: "f67b012deb574287b7fd0c2eb57f4671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb151e4e73443d3a386c8958016b67f" is_pre_election: true
I20260927 22:53:30.539676 13269 raft_consensus.cc:2454] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67b012deb574287b7fd0c2eb57f4671 in term 0.
I20260927 22:53:30.540159 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a5c5b7abaf49378e50afd8b145e646" candidate_uuid: "f67b012deb574287b7fd0c2eb57f4671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" is_pre_election: true
I20260927 22:53:30.540377 13470 leader_election.cc:302] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb151e4e73443d3a386c8958016b67f, f67b012deb574287b7fd0c2eb57f4671; no voters:
I20260927 22:53:30.540632 13402 raft_consensus.cc:2454] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f67b012deb574287b7fd0c2eb57f4671 in term 0.
I20260927 22:53:30.540786 13603 raft_consensus.cc:2781] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:30.540923 13603 raft_consensus.cc:482] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:30.541043 13603 raft_consensus.cc:3037] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:30.543254 13603 raft_consensus.cc:504] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.543877 13603 leader_election.cc:288] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb151e4e73443d3a386c8958016b67f (127.8.204.129:42369), 80adbd3bf7fe4c26a930c29c470dda5c (127.8.204.130:39271)
I20260927 22:53:30.544283 13402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a5c5b7abaf49378e50afd8b145e646" candidate_uuid: "f67b012deb574287b7fd0c2eb57f4671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80adbd3bf7fe4c26a930c29c470dda5c"
I20260927 22:53:30.544283 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a5c5b7abaf49378e50afd8b145e646" candidate_uuid: "f67b012deb574287b7fd0c2eb57f4671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb151e4e73443d3a386c8958016b67f"
I20260927 22:53:30.544473 13269 raft_consensus.cc:3037] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:30.544473 13402 raft_consensus.cc:3037] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:30.547595 13402 raft_consensus.cc:2454] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67b012deb574287b7fd0c2eb57f4671 in term 1.
I20260927 22:53:30.547595 13269 raft_consensus.cc:2454] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f67b012deb574287b7fd0c2eb57f4671 in term 1.
I20260927 22:53:30.547981 13469 leader_election.cc:302] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80adbd3bf7fe4c26a930c29c470dda5c, f67b012deb574287b7fd0c2eb57f4671; no voters:
I20260927 22:53:30.548264 13603 raft_consensus.cc:2781] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:30.548702 13603 raft_consensus.cc:686] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 1 LEADER]: Becoming Leader. State: Replica: f67b012deb574287b7fd0c2eb57f4671, State: Running, Role: LEADER
I20260927 22:53:30.549156 13603 consensus_queue.cc:237] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } }
I20260927 22:53:30.553622 13124 catalog_manager.cc:5495] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 reported cstate change: term changed from 0 to 1, leader changed from <none> to f67b012deb574287b7fd0c2eb57f4671 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "f67b012deb574287b7fd0c2eb57f4671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f67b012deb574287b7fd0c2eb57f4671" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 45517 } health_report { overall_health: HEALTHY } } }
W20260927 22:53:30.557128 13580 tablet.cc:2369] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:30.580596 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.575459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.8.204.131:45517)
0927 22:53:30.575673 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.575680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.575767 (+ 87us) server_negotiation.cc:407] Connection header received
0927 22:53:30.575996 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.576000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.576083 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.576231 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:30.576586 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.577279 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.578586 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.578968 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.579099 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.579615 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:30.580066 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:30.580235 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.580281 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.580353 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:53:30.586660 13448 tablet.cc:2369] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:30.590046 13402 raft_consensus.cc:1261] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 1 FOLLOWER]: Refusing update from remote peer f67b012deb574287b7fd0c2eb57f4671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:30.590157 13269 raft_consensus.cc:1261] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 1 FOLLOWER]: Refusing update from remote peer f67b012deb574287b7fd0c2eb57f4671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:30.591411 13603 consensus_queue.cc:1035] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80adbd3bf7fe4c26a930c29c470dda5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:30.591957 13607 consensus_queue.cc:1035] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb151e4e73443d3a386c8958016b67f" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:53:30.595067 13315 tablet.cc:2369] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:30.609463 13610 mvcc.cc:205] Tried to move back new op lower bound from 7334091204961320960 to 7334091204812853248. Current Snapshot: MvccSnapshot[applied={T|T < 7334091204961320960}]
I20260927 22:53:30.613481 13612 mvcc.cc:205] Tried to move back new op lower bound from 7334091204961320960 to 7334091204812853248. Current Snapshot: MvccSnapshot[applied={T|T < 7334091204961320960}]
I20260927 22:53:30.632591 13124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:30.632879 13124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:30.637226 13124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6330dca416c347d0b348589595f46049: Sending DeleteTablet for 3 replicas of tablet 36a5c5b7abaf49378e50afd8b145e646
I20260927 22:53:30.638469 13249 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a5c5b7abaf49378e50afd8b145e646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:30 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:58992
I20260927 22:53:30.638444 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a5c5b7abaf49378e50afd8b145e646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:30 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34954
I20260927 22:53:30.639366 13621 tablet_replica.cc:330] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: stopping tablet replica
I20260927 22:53:30.639606 13382 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a5c5b7abaf49378e50afd8b145e646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:30 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:55846
I20260927 22:53:30.639974 13621 raft_consensus.cc:2229] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:30.640455 13622 tablet_replica.cc:330] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: stopping tablet replica
I20260927 22:53:30.640503 13621 raft_consensus.cc:2258] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:30.640882 13622 raft_consensus.cc:2229] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:30.641146 13620 tablet_replica.cc:330] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: stopping tablet replica
I20260927 22:53:30.641340 13622 raft_consensus.cc:2258] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:30.641631 13620 raft_consensus.cc:2229] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:30.641738 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13187
I20260927 22:53:30.642390 13620 raft_consensus.cc:2258] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:30.643435 13622 ts_tablet_manager.cc:1905] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:30.644431 13620 ts_tablet_manager.cc:1905] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:30.644791 13621 ts_tablet_manager.cc:1905] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:30.650193 13622 ts_tablet_manager.cc:1918] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:30.650357 13622 log.cc:1196] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/36a5c5b7abaf49378e50afd8b145e646
I20260927 22:53:30.650877 13622 ts_tablet_manager.cc:1939] T 36a5c5b7abaf49378e50afd8b145e646 P 80adbd3bf7fe4c26a930c29c470dda5c: Deleting consensus metadata
I20260927 22:53:30.651752 13621 ts_tablet_manager.cc:1918] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:30.652041 13621 log.cc:1196] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/36a5c5b7abaf49378e50afd8b145e646
I20260927 22:53:30.652595 13621 ts_tablet_manager.cc:1939] T 36a5c5b7abaf49378e50afd8b145e646 P 3fb151e4e73443d3a386c8958016b67f: Deleting consensus metadata
I20260927 22:53:30.653046 13112 catalog_manager.cc:4841] TS 80adbd3bf7fe4c26a930c29c470dda5c (127.8.204.130:39271): tablet 36a5c5b7abaf49378e50afd8b145e646 (table test-table [id=1dacecf1552e4a07b55d26850976b377]) successfully deleted
I20260927 22:53:30.653163 13620 ts_tablet_manager.cc:1918] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:30.653314 13620 log.cc:1196] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/36a5c5b7abaf49378e50afd8b145e646
I20260927 22:53:30.653863 13620 ts_tablet_manager.cc:1939] T 36a5c5b7abaf49378e50afd8b145e646 P f67b012deb574287b7fd0c2eb57f4671: Deleting consensus metadata
I20260927 22:53:30.655409 13113 catalog_manager.cc:4841] TS f67b012deb574287b7fd0c2eb57f4671 (127.8.204.131:45517): tablet 36a5c5b7abaf49378e50afd8b145e646 (table test-table [id=1dacecf1552e4a07b55d26850976b377]) successfully deleted
I20260927 22:53:30.655653 13113 catalog_manager.cc:4841] TS 3fb151e4e73443d3a386c8958016b67f (127.8.204.129:42369): tablet 36a5c5b7abaf49378e50afd8b145e646 (table test-table [id=1dacecf1552e4a07b55d26850976b377]) successfully deleted
Sep 27 22:53:30 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:30.670650 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.643377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.8.204.129:42369)
0927 22:53:30.643577 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.643583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.653847 (+ 10264us) server_negotiation.cc:407] Connection header received
0927 22:53:30.654307 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.654312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.654397 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.654489 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:30.656906 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.658407 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.660199 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.660576 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.660665 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.665103 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:30.665113 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:30.665126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:30.665154 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:30.668074 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.668707 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.668715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.668717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.668838 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.669198 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.669208 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.669210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.669530 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:30.669840 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.670080 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.670308 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:53:30.671355 13309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:30.760092 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13187
I20260927 22:53:30.780740 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13320
Sep 27 22:53:30 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:30.799038 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.781494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.8.204.130:39271)
0927 22:53:30.781654 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.781661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.781724 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:53:30.781958 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.781964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.782037 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.782101 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:30.783837 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.784932 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.786735 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.787553 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.787677 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.792712 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:30.792731 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:30.792753 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:30.792796 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:30.796489 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.797079 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.797088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.797092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.797239 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.797556 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.797561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.797564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.798010 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:30.798294 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.798439 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.798656 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:53:30.799708 13442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:30.887890 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13320
I20260927 22:53:30.910347 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13452
Sep 27 22:53:30 dist-test-slave-40vx krb5kdc[13082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:30.925721 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:30.911115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.8.204.131:45517)
0927 22:53:30.911361 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:30.911366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:30.911387 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:30.911453 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:30.911458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:30.911542 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:30.911653 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:30.912942 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.913758 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.914923 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:30.915571 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:30.915708 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:30.919979 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:30.919990 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:30.920005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:30.920046 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:30.922931 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.923693 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.923700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.923702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.923833 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:30.924248 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:30.924257 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:30.924259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:30.924607 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:30.924930 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:30.925140 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:53:30.925356 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:53:30.926398 13574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:31.046126 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13452
I20260927 22:53:31.064302 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13095
I20260927 22:53:31.065376 13156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:31.171264 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13095
2026-09-27T22:53:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4078 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:31Z Disabled control of system clock
I20260927 22:53:31.235244 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:34513
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:53:31.542852 13631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:31.543115 13631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:31.543193 13631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:31.550479 13631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:31.550598 13631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:31.550635 13631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:31.550684 13631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:31.550726 13631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:31.559821 13631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:34513
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:31.560997 13631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:31.562690 13631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:31.573975 13639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:31.574055 13637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:31.574180 13636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:31.574632 13631 server_base.cc:1034] running on GCE node
I20260927 22:53:31.575321 13631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:31.576561 13631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:31.577749 13631 hybrid_clock.cc:648] HybridClock initialized: now 1790549611577703 us; error 49 us; skew 500 ppm
I20260927 22:53:31.578075 13631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:31.604753 13631 webserver.cc:460] Webserver started at http://127.8.204.190:36147/ using document root <none> and password file <none>
I20260927 22:53:31.605713 13631 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:31.605840 13631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:31.606210 13631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:31.609323 13631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "3ba39d3ea9684e23a1666add7ef79ec7"
format_stamp: "Formatted at 2026-09-27 22:53:31 on dist-test-slave-40vx"
server_key: "34304a333b8ab4ffa58fc54771a907a5"
server_key_iv: "f5ed7c62c825e5d93aebcab1f04d3561"
server_key_version: "encryptionkey@0"
I20260927 22:53:31.610122 13631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "3ba39d3ea9684e23a1666add7ef79ec7"
format_stamp: "Formatted at 2026-09-27 22:53:31 on dist-test-slave-40vx"
server_key: "34304a333b8ab4ffa58fc54771a907a5"
server_key_iv: "f5ed7c62c825e5d93aebcab1f04d3561"
server_key_version: "encryptionkey@0"
I20260927 22:53:31.615345 13631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:53:31.618640 13645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:31.620088 13631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:53:31.620278 13631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "3ba39d3ea9684e23a1666add7ef79ec7"
format_stamp: "Formatted at 2026-09-27 22:53:31 on dist-test-slave-40vx"
server_key: "34304a333b8ab4ffa58fc54771a907a5"
server_key_iv: "f5ed7c62c825e5d93aebcab1f04d3561"
server_key_version: "encryptionkey@0"
I20260927 22:53:31.620486 13631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:31.626355 13631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:31.627066 13631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:31.627393 13631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:31.645375 13631 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:34513
I20260927 22:53:31.645462 13696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:34513 every 8 connection(s)
I20260927 22:53:31.646642 13631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:31.650552 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13631
I20260927 22:53:31.650748 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:31.651300 13697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:31.651484 9010 external_mini_cluster.cc:1428] Setting key 1e1a601911a09ed58fa5ef6d5b832d8f
I20260927 22:53:31.661038 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:31.653421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.8.204.190:34513)
0927 22:53:31.654618 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:31.654634 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:31.654691 (+ 57us) server_negotiation.cc:407] Connection header received
0927 22:53:31.657211 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:31.657244 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:31.657814 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:31.658286 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:31.659369 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:31.659383 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:31.659401 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:31.659419 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:31.659505 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:31.659530 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:31.659775 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:31.659902 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 22:53:31.660184 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":390,"threads_started":1}
I20260927 22:53:31.665450 13697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7: Bootstrap starting.
I20260927 22:53:31.668402 13697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:31.669565 13697 log.cc:824] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:31.672190 13697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7: No bootstrap required, opened a new log
I20260927 22:53:31.679183 13697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 34513 } }
I20260927 22:53:31.679636 13697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:31.679738 13697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba39d3ea9684e23a1666add7ef79ec7, State: Initialized, Role: FOLLOWER
I20260927 22:53:31.680327 13697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 34513 } }
I20260927 22:53:31.680557 13697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:31.680701 13697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:31.680873 13697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:31.682901 13697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 34513 } }
I20260927 22:53:31.683451 13697 leader_election.cc:302] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ba39d3ea9684e23a1666add7ef79ec7; no voters:
I20260927 22:53:31.684196 13697 leader_election.cc:288] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:31.684361 13702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:31.685115 13702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba39d3ea9684e23a1666add7ef79ec7, State: Running, Role: LEADER
I20260927 22:53:31.685642 13702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 34513 } }
I20260927 22:53:31.686507 13697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:31.688458 13703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 34513 } } }
I20260927 22:53:31.688793 13703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:31.689752 13704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ba39d3ea9684e23a1666add7ef79ec7. Latest consensus state: current_term: 1 leader_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ba39d3ea9684e23a1666add7ef79ec7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 34513 } } }
I20260927 22:53:31.689999 13709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:31.690069 13704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:31.696224 13709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:31.705624 13709 catalog_manager.cc:1347] Generated new cluster ID: 1dc826b96ad34c55ae7ff1720b32d81a
I20260927 22:53:31.705719 13709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:31.743400 13709 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:31.743815 13709 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:31.758129 13709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7: Generated new TSK 0
I20260927 22:53:31.758970 13709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:53:31.776468 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:34513
--builtin_ntp_servers=127.8.204.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:53:32.118716 13721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:32.118988 13721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:32.119083 13721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:32.126513 13721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:32.126621 13721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:32.126732 13721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:32.135998 13721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:32.137248 13721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:32.138947 13721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:32.151623 13729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:32.151665 13727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:32.151595 13726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:32.152402 13721 server_base.cc:1034] running on GCE node
I20260927 22:53:32.152956 13721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:32.153842 13721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:32.155022 13721 hybrid_clock.cc:648] HybridClock initialized: now 1790549612154969 us; error 52 us; skew 500 ppm
I20260927 22:53:32.155382 13721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:32.157475 13721 webserver.cc:460] Webserver started at http://127.8.204.129:42863/ using document root <none> and password file <none>
I20260927 22:53:32.158177 13721 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:32.158288 13721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:32.158602 13721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:32.161414 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "5203f03ee7a67aff8d9674922e698f79"
server_key_iv: "cf4072e526698c32512b3fe477d3acd1"
server_key_version: "encryptionkey@0"
I20260927 22:53:32.162213 13721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "5203f03ee7a67aff8d9674922e698f79"
server_key_iv: "cf4072e526698c32512b3fe477d3acd1"
server_key_version: "encryptionkey@0"
I20260927 22:53:32.167328 13721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:53:32.170908 13735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:32.172379 13721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:53:32.172551 13721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "5203f03ee7a67aff8d9674922e698f79"
server_key_iv: "cf4072e526698c32512b3fe477d3acd1"
server_key_version: "encryptionkey@0"
I20260927 22:53:32.172785 13721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:32.178417 13721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:32.179242 13721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:32.179585 13721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:32.180473 13721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:32.181824 13721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:32.181952 13721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:32.182083 13721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:32.182168 13721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:32.214759 13721 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:40117
I20260927 22:53:32.214847 13847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:40117 every 8 connection(s)
I20260927 22:53:32.216181 13721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:32.217391 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13721
I20260927 22:53:32.217568 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:32.218374 9010 external_mini_cluster.cc:1428] Setting key 7829da14cd8c50d5a7bc5eb80443a553
I20260927 22:53:32.221120 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:34513
--builtin_ntp_servers=127.8.204.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:53:32.225323 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:32.218988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:46281 (local address 127.8.204.190:34513)
0927 22:53:32.220174 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:32.220180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:32.221132 (+ 952us) server_negotiation.cc:407] Connection header received
0927 22:53:32.222794 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:32.222803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:32.222871 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:32.222989 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:32.224402 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:32.224411 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:32.224418 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:32.224441 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:32.224497 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:32.224515 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:32.224651 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:32.224797 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:53:32.224859 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":958,"threads_started":1}
I20260927 22:53:32.226082 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:32.219819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:34513 (local address 127.8.204.129:46281)
0927 22:53:32.220959 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:32.221011 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:32.222516 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:32.223185 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:32.223193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:32.223347 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:32.224035 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:32.224038 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:32.224077 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:32.224228 (+ 151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:32.224616 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:32.224623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:32.224628 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:53:32.224832 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 22:53:32.225028 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":353,"threads_started":1}
I20260927 22:53:32.227731 13848 heartbeater.cc:347] Connected to a master server at 127.8.204.190:34513
I20260927 22:53:32.228055 13848 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:32.228644 13848 heartbeater.cc:511] Master 127.8.204.190:34513 requested a full tablet report, sending...
I20260927 22:53:32.230823 13662 ts_manager.cc:194] Registered new tserver with Master: 8c5fb6ca197f4bd39b7ad4ed828dad17 (127.8.204.129:40117)
W20260927 22:53:32.538087 13852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:32.538365 13852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:32.538452 13852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:32.545491 13852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:32.545585 13852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:32.545689 13852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:32.554893 13852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:32.556106 13852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:32.557739 13852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:32.570472 13857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:32.570449 13860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:32.570423 13858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:32.570992 13852 server_base.cc:1034] running on GCE node
I20260927 22:53:32.571676 13852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:32.572606 13852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:32.573794 13852 hybrid_clock.cc:648] HybridClock initialized: now 1790549612573749 us; error 45 us; skew 500 ppm
I20260927 22:53:32.574128 13852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:32.576457 13852 webserver.cc:460] Webserver started at http://127.8.204.130:39675/ using document root <none> and password file <none>
I20260927 22:53:32.577162 13852 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:32.577281 13852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:32.577596 13852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:32.580463 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "5f2dcb552f444d1aa11d6d1191881637"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "9f146daac034b6b3685876a0605641a2"
server_key_iv: "4dd3d27695a4787cbb0fd9b92762171f"
server_key_version: "encryptionkey@0"
I20260927 22:53:32.581197 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "5f2dcb552f444d1aa11d6d1191881637"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "9f146daac034b6b3685876a0605641a2"
server_key_iv: "4dd3d27695a4787cbb0fd9b92762171f"
server_key_version: "encryptionkey@0"
I20260927 22:53:32.586155 13852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:53:32.589614 13866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:32.590837 13852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260927 22:53:32.591001 13852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "5f2dcb552f444d1aa11d6d1191881637"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "9f146daac034b6b3685876a0605641a2"
server_key_iv: "4dd3d27695a4787cbb0fd9b92762171f"
server_key_version: "encryptionkey@0"
I20260927 22:53:32.591228 13852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:32.596853 13852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:32.597587 13852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:32.597851 13852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:32.598680 13852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:32.599954 13852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:32.600090 13852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:32.600217 13852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:32.600318 13852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:32.631594 13852 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38951
I20260927 22:53:32.631633 13978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38951 every 8 connection(s)
I20260927 22:53:32.632865 13852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:32.640357 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:32.635315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:51403 (local address 127.8.204.190:34513)
0927 22:53:32.635486 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:32.635491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:32.636887 (+ 1396us) server_negotiation.cc:407] Connection header received
0927 22:53:32.638442 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:32.638447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:32.638496 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:32.638579 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:32.639820 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:32.639824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:32.639827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:32.639840 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:32.639865 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:32.639875 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:32.639959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:32.640114 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 22:53:32.640160 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:53:32.641198 13981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:32.635849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:34513 (local address 127.8.204.130:51403)
0927 22:53:32.636738 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:32.636785 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:32.638217 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:32.638769 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:32.638777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:32.638860 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:32.639561 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:32.639563 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:32.639599 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:32.639679 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:32.639955 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:32.639961 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:32.639966 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:53:32.640159 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 22:53:32.640332 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":261,"threads_started":1}
I20260927 22:53:32.641252 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13852
I20260927 22:53:32.641403 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:32.641954 9010 external_mini_cluster.cc:1428] Setting key b53e4780ea1e9c9942725c8a4a7c6b88
I20260927 22:53:32.642596 13979 heartbeater.cc:347] Connected to a master server at 127.8.204.190:34513
I20260927 22:53:32.642882 13979 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:32.643476 13979 heartbeater.cc:511] Master 127.8.204.190:34513 requested a full tablet report, sending...
I20260927 22:53:32.644832 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:34513
--builtin_ntp_servers=127.8.204.148:34461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:53:32.644826 13662 ts_manager.cc:194] Registered new tserver with Master: 5f2dcb552f444d1aa11d6d1191881637 (127.8.204.130:38951)
W20260927 22:53:32.958734 13982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:32.959059 13982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:32.959218 13982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:32.966799 13982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:32.966930 13982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:32.967111 13982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:32.976277 13982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:32.977551 13982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:32.979354 13982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:32.992188 13988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:32.992202 13987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:32.992216 13990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:32.992931 13982 server_base.cc:1034] running on GCE node
I20260927 22:53:32.993556 13982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:32.994500 13982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:32.995728 13982 hybrid_clock.cc:648] HybridClock initialized: now 1790549612995670 us; error 47 us; skew 500 ppm
I20260927 22:53:32.996032 13982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:32.998137 13982 webserver.cc:460] Webserver started at http://127.8.204.131:35957/ using document root <none> and password file <none>
I20260927 22:53:32.998783 13982 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:32.998903 13982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:32.999243 13982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:33.001964 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "9a45a98a3cfb49839d18fbb539ff2e46"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "55729a877f5bfb4c982d7d9dc542cbf9"
server_key_iv: "efe8eed953f4e061da436e751a6ed1d8"
server_key_version: "encryptionkey@0"
I20260927 22:53:33.002782 13982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "9a45a98a3cfb49839d18fbb539ff2e46"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "55729a877f5bfb4c982d7d9dc542cbf9"
server_key_iv: "efe8eed953f4e061da436e751a6ed1d8"
server_key_version: "encryptionkey@0"
I20260927 22:53:33.007834 13982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:53:33.011452 13996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:33.012908 13982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:33.013074 13982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "9a45a98a3cfb49839d18fbb539ff2e46"
format_stamp: "Formatted at 2026-09-27 22:53:32 on dist-test-slave-40vx"
server_key: "55729a877f5bfb4c982d7d9dc542cbf9"
server_key_iv: "efe8eed953f4e061da436e751a6ed1d8"
server_key_version: "encryptionkey@0"
I20260927 22:53:33.013320 13982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:33.018988 13982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:33.019791 13982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:33.020036 13982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:33.020905 13982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:33.022223 13982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:33.022338 13982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:33.022464 13982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:33.022557 13982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:33.054368 13982 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:33699
I20260927 22:53:33.054419 14108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:33699 every 8 connection(s)
I20260927 22:53:33.055794 13982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:33.063685 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 13982
I20260927 22:53:33.063828 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.058362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:42791 (local address 127.8.204.190:34513)
0927 22:53:33.058539 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.058544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.059944 (+ 1400us) server_negotiation.cc:407] Connection header received
0927 22:53:33.061591 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.061597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.061670 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.061779 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.063039 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.063046 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.063051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.063069 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.063108 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.063120 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.063241 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.063481 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.063557 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:53:33.063848 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:33.064311 9010 external_mini_cluster.cc:1428] Setting key 7f58b0ad5571d166b20757b7ef68e1d3
I20260927 22:53:33.064733 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.058893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:34513 (local address 127.8.204.131:42791)
0927 22:53:33.059793 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:33.059833 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:33.061326 (+ 1493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:33.061989 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:33.061997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:33.062091 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:33.062761 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:33.062764 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:33.062798 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:33.062876 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:33.063214 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:33.063223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:33.063230 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:53:33.063549 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:53:33.063779 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":251,"threads_started":1}
I20260927 22:53:33.065997 14109 heartbeater.cc:347] Connected to a master server at 127.8.204.190:34513
I20260927 22:53:33.066282 14109 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:33.066931 14109 heartbeater.cc:511] Master 127.8.204.190:34513 requested a full tablet report, sending...
I20260927 22:53:33.068389 13662 ts_manager.cc:194] Registered new tserver with Master: 9a45a98a3cfb49839d18fbb539ff2e46 (127.8.204.131:33699)
I20260927 22:53:33.079641 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:53:33.086061 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.083822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.8.204.190:34513)
0927 22:53:33.084038 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.084048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.084434 (+ 386us) server_negotiation.cc:407] Connection header received
0927 22:53:33.084526 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.084530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.084580 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.084661 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.085556 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.085559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.085562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.085574 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.085597 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.085606 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.085691 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.085724 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.085762 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:53:33.089875 13662 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:33.096794 13662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:33.129168 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.108171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:33699 (local address 127.0.0.1:41220)
0927 22:53:33.120523 (+ 12352us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:33.120539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:33.120624 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:33.125984 (+ 5360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:33.125997 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:33.126124 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:33.127101 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:33.127107 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:33.127147 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:33.127263 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:33.128194 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:33.128210 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:33.128223 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:53:33.128435 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 22:53:33.128559 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12211,"thread_start_us":12092,"threads_started":1}
I20260927 22:53:33.130721 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.108675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.8.204.131:33699)
0927 22:53:33.125281 (+ 16606us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.125293 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.125340 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:53:33.125438 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.125450 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.125735 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.125922 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.127531 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.127556 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.127577 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.127604 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.127688 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.127714 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.130210 (+ 2496us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.130284 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.130386 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16383,"thread_start_us":18447,"threads_started":1}
I20260927 22:53:33.132095 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.109219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.8.204.129:40117)
0927 22:53:33.128420 (+ 19201us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.128430 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.128460 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:53:33.128525 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.128532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.128714 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.128890 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.130297 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.130305 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.130318 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.130334 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.130388 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.130411 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.130540 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.131373 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.131475 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19027,"thread_start_us":18909,"threads_started":1}
I20260927 22:53:33.132256 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.108668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40117 (local address 127.0.0.1:40526)
0927 22:53:33.127748 (+ 19080us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:33.127780 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:33.127896 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:33.128877 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:33.128882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:33.128918 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:33.130065 (+ 1147us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:33.130068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:33.130085 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:33.130130 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:33.130593 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:33.130597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:33.130602 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:53:33.130743 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:53:33.131259 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18946,"thread_start_us":18822,"threads_started":1}
I20260927 22:53:33.132790 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.108261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.8.204.130:38951)
0927 22:53:33.130117 (+ 21856us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.130125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.130154 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:53:33.130220 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.130229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.130441 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.130581 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.131530 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.131538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.131550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.131571 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.131615 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.131632 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.131787 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.132349 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.132458 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21690,"thread_start_us":21570,"threads_started":1}
I20260927 22:53:33.133296 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.108037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38951 (local address 127.0.0.1:37414)
0927 22:53:33.119968 (+ 11931us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:33.120035 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:33.120221 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:33.130619 (+ 10398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:33.130625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:33.130649 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:33.131309 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:33.131312 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:33.131327 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:33.131365 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:33.131846 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:33.131851 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:33.131856 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:53:33.131995 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:53:33.132046 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11566,"thread_start_us":12339,"threads_started":1}
I20260927 22:53:33.134680 14044 tablet_service.cc:1467] Processing CreateTablet for tablet d6f439d0ab4342529ddfc7e942c56063 (DEFAULT_TABLE table=test-table [id=f30eb235190b48c1852cbea5dfa29057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:33.135856 13783 tablet_service.cc:1467] Processing CreateTablet for tablet d6f439d0ab4342529ddfc7e942c56063 (DEFAULT_TABLE table=test-table [id=f30eb235190b48c1852cbea5dfa29057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:33.136513 14044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f439d0ab4342529ddfc7e942c56063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:33.136384 13914 tablet_service.cc:1467] Processing CreateTablet for tablet d6f439d0ab4342529ddfc7e942c56063 (DEFAULT_TABLE table=test-table [id=f30eb235190b48c1852cbea5dfa29057]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:33.137166 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f439d0ab4342529ddfc7e942c56063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:33.137555 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f439d0ab4342529ddfc7e942c56063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:33.148710 14127 tablet_bootstrap.cc:492] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Bootstrap starting.
I20260927 22:53:33.151363 14128 tablet_bootstrap.cc:492] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Bootstrap starting.
I20260927 22:53:33.152391 14129 tablet_bootstrap.cc:492] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Bootstrap starting.
I20260927 22:53:33.153317 14127 tablet_bootstrap.cc:654] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:33.154704 14127 log.cc:824] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:33.155416 14129 tablet_bootstrap.cc:654] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:33.155478 14128 tablet_bootstrap.cc:654] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:33.156497 14129 log.cc:824] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:33.156639 14128 log.cc:824] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:33.157958 14127 tablet_bootstrap.cc:492] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: No bootstrap required, opened a new log
I20260927 22:53:33.158259 14127 ts_tablet_manager.cc:1397] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:53:33.159443 14128 tablet_bootstrap.cc:492] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: No bootstrap required, opened a new log
I20260927 22:53:33.159657 14129 tablet_bootstrap.cc:492] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: No bootstrap required, opened a new log
I20260927 22:53:33.159700 14128 ts_tablet_manager.cc:1397] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:53:33.159927 14129 ts_tablet_manager.cc:1397] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260927 22:53:33.167168 14128 raft_consensus.cc:348] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.167635 14128 raft_consensus.cc:374] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:33.167726 14128 raft_consensus.cc:729] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5fb6ca197f4bd39b7ad4ed828dad17, State: Initialized, Role: FOLLOWER
I20260927 22:53:33.168382 14128 consensus_queue.cc:260] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.169539 13848 heartbeater.cc:503] Master 127.8.204.190:34513 was elected leader, sending a full tablet report...
I20260927 22:53:33.170104 14128 ts_tablet_manager.cc:1428] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:53:33.169173 14127 raft_consensus.cc:348] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.170245 14129 raft_consensus.cc:348] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.170594 14127 raft_consensus.cc:374] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:33.170658 14129 raft_consensus.cc:374] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:33.170701 14127 raft_consensus.cc:729] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2dcb552f444d1aa11d6d1191881637, State: Initialized, Role: FOLLOWER
I20260927 22:53:33.170756 14129 raft_consensus.cc:729] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a45a98a3cfb49839d18fbb539ff2e46, State: Initialized, Role: FOLLOWER
I20260927 22:53:33.171367 14127 consensus_queue.cc:260] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.171470 14129 consensus_queue.cc:260] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.174316 14109 heartbeater.cc:503] Master 127.8.204.190:34513 was elected leader, sending a full tablet report...
I20260927 22:53:33.174436 13979 heartbeater.cc:503] Master 127.8.204.190:34513 was elected leader, sending a full tablet report...
I20260927 22:53:33.174727 14129 ts_tablet_manager.cc:1428] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 22:53:33.175035 14127 ts_tablet_manager.cc:1428] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260927 22:53:33.182655 14135 raft_consensus.cc:482] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:33.182940 14135 raft_consensus.cc:504] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.184612 14135 leader_election.cc:288] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a45a98a3cfb49839d18fbb539ff2e46 (127.8.204.131:33699), 8c5fb6ca197f4bd39b7ad4ed828dad17 (127.8.204.129:40117)
I20260927 22:53:33.188666 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.185332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:57005 (local address 127.8.204.129:40117)
0927 22:53:33.185540 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.185547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.186367 (+ 820us) server_negotiation.cc:407] Connection header received
0927 22:53:33.186790 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.186798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.186868 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.186976 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.188066 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.188071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.188076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.188099 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.188126 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.188137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.188252 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.188329 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.188390 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:53:33.188831 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.184948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40117 (local address 127.8.204.130:57005)
0927 22:53:33.185847 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:33.185869 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:33.186479 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:33.187032 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:33.187037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:33.187059 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:33.187878 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:33.187881 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:33.187894 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:33.187929 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:33.188225 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:33.188229 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:33.188234 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:53:33.188379 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:53:33.188426 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":599,"threads_started":1}
I20260927 22:53:33.188973 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.184948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:55411 (local address 127.8.204.131:33699)
0927 22:53:33.185185 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.185193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.187212 (+ 2019us) server_negotiation.cc:407] Connection header received
0927 22:53:33.187397 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.187402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.187454 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.187525 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.188404 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.188410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.188414 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.188432 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.188468 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.188486 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.188614 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.188672 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.188729 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:53:33.189486 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.184966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:33699 (local address 127.8.204.130:55411)
0927 22:53:33.187105 (+ 2139us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:33.187133 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:33.187250 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:33.187554 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:33.187559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:33.187583 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:33.188216 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:53:33.188218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:33.188229 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:53:33.188262 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:33.188579 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:33.188583 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:33.188588 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:53:33.188714 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 22:53:33.189063 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2024,"thread_start_us":1833,"threads_started":1}
I20260927 22:53:33.189983 13803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f439d0ab4342529ddfc7e942c56063" candidate_uuid: "5f2dcb552f444d1aa11d6d1191881637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" is_pre_election: true
I20260927 22:53:33.190378 14064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f439d0ab4342529ddfc7e942c56063" candidate_uuid: "5f2dcb552f444d1aa11d6d1191881637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" is_pre_election: true
I20260927 22:53:33.190570 13803 raft_consensus.cc:2454] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f2dcb552f444d1aa11d6d1191881637 in term 0.
I20260927 22:53:33.190899 14064 raft_consensus.cc:2454] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f2dcb552f444d1aa11d6d1191881637 in term 0.
I20260927 22:53:33.191263 13867 leader_election.cc:302] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f2dcb552f444d1aa11d6d1191881637, 8c5fb6ca197f4bd39b7ad4ed828dad17; no voters:
I20260927 22:53:33.191686 14135 raft_consensus.cc:2781] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:33.191834 14135 raft_consensus.cc:482] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:33.191992 14135 raft_consensus.cc:3037] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:33.194247 14135 raft_consensus.cc:504] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.194875 14135 leader_election.cc:288] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [CANDIDATE]: Term 1 election: Requested vote from peers 9a45a98a3cfb49839d18fbb539ff2e46 (127.8.204.131:33699), 8c5fb6ca197f4bd39b7ad4ed828dad17 (127.8.204.129:40117)
I20260927 22:53:33.195238 14064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f439d0ab4342529ddfc7e942c56063" candidate_uuid: "5f2dcb552f444d1aa11d6d1191881637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a45a98a3cfb49839d18fbb539ff2e46"
I20260927 22:53:33.195318 13803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f439d0ab4342529ddfc7e942c56063" candidate_uuid: "5f2dcb552f444d1aa11d6d1191881637" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17"
I20260927 22:53:33.195505 14064 raft_consensus.cc:3037] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:33.195549 13803 raft_consensus.cc:3037] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:33.197670 14064 raft_consensus.cc:2454] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f2dcb552f444d1aa11d6d1191881637 in term 1.
I20260927 22:53:33.197692 13803 raft_consensus.cc:2454] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f2dcb552f444d1aa11d6d1191881637 in term 1.
I20260927 22:53:33.198060 13867 leader_election.cc:302] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f2dcb552f444d1aa11d6d1191881637, 8c5fb6ca197f4bd39b7ad4ed828dad17; no voters:
I20260927 22:53:33.198321 14135 raft_consensus.cc:2781] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:33.198720 14135 raft_consensus.cc:686] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 1 LEADER]: Becoming Leader. State: Replica: 5f2dcb552f444d1aa11d6d1191881637, State: Running, Role: LEADER
I20260927 22:53:33.199134 14135 consensus_queue.cc:237] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } }
I20260927 22:53:33.203593 13661 catalog_manager.cc:5495] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f2dcb552f444d1aa11d6d1191881637 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "5f2dcb552f444d1aa11d6d1191881637" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f2dcb552f444d1aa11d6d1191881637" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 } health_report { overall_health: UNKNOWN } } }
W20260927 22:53:33.218334 13849 tablet.cc:2369] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:33.230615 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.228535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.8.204.130:38951)
0927 22:53:33.228746 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.228752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.228771 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:53:33.228946 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.228950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.229021 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.229131 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.230046 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.230051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.230055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.230072 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.230113 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.230124 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.230253 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.230328 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.230383 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:53:33.238304 14064 raft_consensus.cc:1261] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 1 FOLLOWER]: Refusing update from remote peer 5f2dcb552f444d1aa11d6d1191881637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:33.238304 13803 raft_consensus.cc:1261] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 1 FOLLOWER]: Refusing update from remote peer 5f2dcb552f444d1aa11d6d1191881637: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:33.239353 14138 consensus_queue.cc:1035] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5fb6ca197f4bd39b7ad4ed828dad17" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:33.239696 14135 consensus_queue.cc:1035] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a45a98a3cfb49839d18fbb539ff2e46" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:33.264659 13660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:33.264927 13660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:33.268527 13660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ba39d3ea9684e23a1666add7ef79ec7: Sending DeleteTablet for 3 replicas of tablet d6f439d0ab4342529ddfc7e942c56063
I20260927 22:53:33.269632 14044 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f439d0ab4342529ddfc7e942c56063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:33 UTC) from {username='slave'} at 127.0.0.1:41220
I20260927 22:53:33.269624 13914 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f439d0ab4342529ddfc7e942c56063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:33 UTC) from {username='slave'} at 127.0.0.1:37414
I20260927 22:53:33.269956 13783 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f439d0ab4342529ddfc7e942c56063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:33 UTC) from {username='slave'} at 127.0.0.1:40526
I20260927 22:53:33.272233 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13721
I20260927 22:53:33.285029 14153 tablet_replica.cc:330] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: stopping tablet replica
I20260927 22:53:33.285552 14153 raft_consensus.cc:2229] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:33.286105 14153 raft_consensus.cc:2258] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:33.288683 14153 ts_tablet_manager.cc:1905] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:33.297178 14153 ts_tablet_manager.cc:1918] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:33.297341 14153 log.cc:1196] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/d6f439d0ab4342529ddfc7e942c56063
I20260927 22:53:33.297884 14153 ts_tablet_manager.cc:1939] T d6f439d0ab4342529ddfc7e942c56063 P 8c5fb6ca197f4bd39b7ad4ed828dad17: Deleting consensus metadata
I20260927 22:53:33.299747 13646 catalog_manager.cc:4841] TS 8c5fb6ca197f4bd39b7ad4ed828dad17 (127.8.204.129:40117): tablet d6f439d0ab4342529ddfc7e942c56063 (table test-table [id=f30eb235190b48c1852cbea5dfa29057]) successfully deleted
I20260927 22:53:33.300518 14152 tablet_replica.cc:330] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: stopping tablet replica
I20260927 22:53:33.300990 14152 raft_consensus.cc:2229] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:33.301504 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.274030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.8.204.129:40117)
0927 22:53:33.274247 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.274256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.298584 (+ 24328us) server_negotiation.cc:407] Connection header received
0927 22:53:33.299042 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.299048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.299146 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.299218 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.300730 (+ 1512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.300735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.300739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.300759 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.300793 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.300808 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.300904 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.301159 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.301217 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:53:33.301746 14152 raft_consensus.cc:2258] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:33.302532 14151 tablet_replica.cc:330] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: stopping tablet replica
I20260927 22:53:33.302919 13843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:33.302963 14151 raft_consensus.cc:2229] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:33.303480 14151 raft_consensus.cc:2258] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:33.304432 14152 ts_tablet_manager.cc:1905] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:33.305493 14151 ts_tablet_manager.cc:1905] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:33.311770 14152 ts_tablet_manager.cc:1918] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:33.312022 14152 log.cc:1196] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/d6f439d0ab4342529ddfc7e942c56063
I20260927 22:53:33.312213 14151 ts_tablet_manager.cc:1918] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:33.312372 14151 log.cc:1196] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/d6f439d0ab4342529ddfc7e942c56063
I20260927 22:53:33.312844 14151 ts_tablet_manager.cc:1939] T d6f439d0ab4342529ddfc7e942c56063 P 9a45a98a3cfb49839d18fbb539ff2e46: Deleting consensus metadata
I20260927 22:53:33.312844 14152 ts_tablet_manager.cc:1939] T d6f439d0ab4342529ddfc7e942c56063 P 5f2dcb552f444d1aa11d6d1191881637: Deleting consensus metadata
I20260927 22:53:33.314563 13649 catalog_manager.cc:4841] TS 9a45a98a3cfb49839d18fbb539ff2e46 (127.8.204.131:33699): tablet d6f439d0ab4342529ddfc7e942c56063 (table test-table [id=f30eb235190b48c1852cbea5dfa29057]) successfully deleted
I20260927 22:53:33.314855 13647 catalog_manager.cc:4841] TS 5f2dcb552f444d1aa11d6d1191881637 (127.8.204.130:38951): tablet d6f439d0ab4342529ddfc7e942c56063 (table test-table [id=f30eb235190b48c1852cbea5dfa29057]) successfully deleted
I20260927 22:53:33.403059 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13721
I20260927 22:53:33.420809 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13852
I20260927 22:53:33.423727 14123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.421705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37418 (local address 127.8.204.130:38951)
0927 22:53:33.421877 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.421881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.421898 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:53:33.422023 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.422027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.422087 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.422169 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.423055 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.423068 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.423072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.423086 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.423117 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.423128 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.423231 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.423367 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.423411 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:53:33.424539 13974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:33.533107 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13852
I20260927 22:53:33.550827 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13982
I20260927 22:53:33.553419 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:33.551611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.8.204.131:33699)
0927 22:53:33.551780 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:33.551784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:33.551873 (+ 89us) server_negotiation.cc:407] Connection header received
0927 22:53:33.552036 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:33.552040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:33.552112 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:33.552196 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:33.552977 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:33.552981 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:33.552984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:33.552997 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:33.553025 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:33.553036 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:33.553135 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:33.553168 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 22:53:33.553210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:53:33.553974 14104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:33.640810 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13982
I20260927 22:53:33.661440 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 13631
I20260927 22:53:33.662552 13692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:33.767498 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 13631
2026-09-27T22:53:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2595 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:53:33.809134 9010 mini_oidc.cc:152] Starting JWKS server
I20260927 22:53:33.813434 9010 webserver.cc:460] Webserver started at https://127.0.0.1:37475/ using document root <none> and password file <none>
I20260927 22:53:33.813825 9010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:53:33.820164 9010 webserver.cc:460] Webserver started at http://127.0.0.1:32901/ using document root <none> and password file <none>
2026-09-27T22:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:33Z Disabled control of system clock
I20260927 22:53:33.845005 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39215
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:39215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37475/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/testchainca.cert with env {}
W20260927 22:53:34.156862 14164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:34.157196 14164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:34.157364 14164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:34.164423 14164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:34.164565 14164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:34.164659 14164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:34.164762 14164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:34.164849 14164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:53:34.164937 14164 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:53:34.164994 14164 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37475/jwks.json
I20260927 22:53:34.174583 14164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:39215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37475/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:34.175861 14164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:34.177127 14164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:34.188704 14169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:34.188802 14170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:34.188756 14172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:34.189944 14164 server_base.cc:1034] running on GCE node
I20260927 22:53:34.190716 14164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:34.192119 14164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:34.193338 14164 hybrid_clock.cc:648] HybridClock initialized: now 1790549614193307 us; error 45 us; skew 500 ppm
I20260927 22:53:34.193676 14164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:34.195896 14164 webserver.cc:460] Webserver started at http://127.8.204.190:39193/ using document root <none> and password file <none>
I20260927 22:53:34.196576 14164 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:34.196715 14164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:34.197021 14164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:34.199909 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "73beca62eaea422596df7f91f74652d8"
format_stamp: "Formatted at 2026-09-27 22:53:34 on dist-test-slave-40vx"
server_key: "31da389ee1aa5c4b753430315be302ee"
server_key_iv: "8f27a750304a553e429c70cafb3bd55f"
server_key_version: "encryptionkey@0"
I20260927 22:53:34.200668 14164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "73beca62eaea422596df7f91f74652d8"
format_stamp: "Formatted at 2026-09-27 22:53:34 on dist-test-slave-40vx"
server_key: "31da389ee1aa5c4b753430315be302ee"
server_key_iv: "8f27a750304a553e429c70cafb3bd55f"
server_key_version: "encryptionkey@0"
I20260927 22:53:34.205394 14164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:53:34.208514 14178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:34.209755 14164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 22:53:34.209923 14164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "73beca62eaea422596df7f91f74652d8"
format_stamp: "Formatted at 2026-09-27 22:53:34 on dist-test-slave-40vx"
server_key: "31da389ee1aa5c4b753430315be302ee"
server_key_iv: "8f27a750304a553e429c70cafb3bd55f"
server_key_version: "encryptionkey@0"
I20260927 22:53:34.210132 14164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:34.236508 14164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:34.237269 14164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:34.237563 14164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:34.255147 14164 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:39215
I20260927 22:53:34.255164 14229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:39215 every 8 connection(s)
I20260927 22:53:34.256456 14164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:34.259977 14230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:34.260295 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14164
I20260927 22:53:34.260495 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:34.261117 9010 external_mini_cluster.cc:1428] Setting key 1bf012b4cb8076615f1e1a1b71c928c4
I20260927 22:53:34.274791 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8: Bootstrap starting.
I20260927 22:53:34.276911 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:34.263082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.8.204.190:39215)
0927 22:53:34.264184 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:34.264198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:34.264245 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:53:34.266635 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:34.266662 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:34.267293 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:34.267748 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:34.268270 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.270088 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.271508 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.272627 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.272818 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:34.274034 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:34.274044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:34.274063 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:34.274822 (+ 759us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:34.274929 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:34.274955 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:34.275061 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:34.275255 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:53:34.275619 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":376,"threads_started":1}
I20260927 22:53:34.278398 14230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:34.279518 14230 log.cc:824] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:34.280639 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:53:34.284101 14230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8: No bootstrap required, opened a new log
I20260927 22:53:34.290882 14230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73beca62eaea422596df7f91f74652d8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39215 } }
I20260927 22:53:34.291206 14230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:34.291321 14230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73beca62eaea422596df7f91f74652d8, State: Initialized, Role: FOLLOWER
I20260927 22:53:34.291803 14230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73beca62eaea422596df7f91f74652d8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39215 } }
I20260927 22:53:34.292069 14230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:34.292174 14230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:34.292284 14230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:34.294513 14230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73beca62eaea422596df7f91f74652d8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39215 } }
I20260927 22:53:34.294939 14230 leader_election.cc:302] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73beca62eaea422596df7f91f74652d8; no voters:
I20260927 22:53:34.295478 14230 leader_election.cc:288] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:34.295634 14235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:34.296322 14235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [term 1 LEADER]: Becoming Leader. State: Replica: 73beca62eaea422596df7f91f74652d8, State: Running, Role: LEADER
I20260927 22:53:34.296854 14235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73beca62eaea422596df7f91f74652d8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39215 } }
I20260927 22:53:34.297489 14230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:34.299681 14236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73beca62eaea422596df7f91f74652d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73beca62eaea422596df7f91f74652d8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39215 } } }
I20260927 22:53:34.300139 14236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:34.300655 14237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73beca62eaea422596df7f91f74652d8. Latest consensus state: current_term: 1 leader_uuid: "73beca62eaea422596df7f91f74652d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73beca62eaea422596df7f91f74652d8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39215 } } }
I20260927 22:53:34.302266 14237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:34.303465 14244 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:34.307538 14244 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:34.315450 14244 catalog_manager.cc:1347] Generated new cluster ID: 925941faa4af456f8d32843e42872919
I20260927 22:53:34.315554 14244 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:34.333097 14244 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:34.334460 14244 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:34.341562 14244 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73beca62eaea422596df7f91f74652d8: Generated new TSK 0
I20260927 22:53:34.342372 14244 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:53:34.383399 14260 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:53:34.383596 14260 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:53:34.384256 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:34.362455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.8.204.190:39215)
0927 22:53:34.362649 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:34.362656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:34.363181 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:53:34.363523 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:34.363529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:34.363646 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:34.363799 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:53:34.364315 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.365423 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.366766 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.367485 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.367624 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:34.368077 (+ 453us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:53:34.383356 (+ 15279us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:53:34.383929 (+ 573us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 22:53:34.384370 14233 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:53:34.385741 14259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:34.362395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39215 (local address 127.0.0.1:43436)
0927 22:53:34.363078 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:34.363097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:34.363234 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:34.363830 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:34.363833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:34.363842 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:53:34.364170 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:34.364178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.365548 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.365551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:34.366632 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:34.366642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.367651 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.367655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:34.367769 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:34.367825 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:53:34.383715 (+ 15890us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:53:34.384001 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.190:39215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":364,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.204.190:39215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37475/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:53:34.387908 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14164
I20260927 22:53:34.388947 14225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:34.479017 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14164
2026-09-27T22:53:34Z chronyd exiting
I20260927 22:53:34.528647 9010 test_util.cc:175] -----------------------------------------------
I20260927 22:53:34.528777 9010 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790549572925352-9010-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (725 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:53:34.532423 9010 mini_oidc.cc:152] Starting JWKS server
I20260927 22:53:34.534183 9010 webserver.cc:460] Webserver started at https://127.0.0.1:37259/ using document root <none> and password file <none>
I20260927 22:53:34.534401 9010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:53:34.534984 9010 webserver.cc:460] Webserver started at http://127.0.0.1:42651/ using document root <none> and password file <none>
2026-09-27T22:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:34Z Disabled control of system clock
I20260927 22:53:34.559405 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45197
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:45197
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37259/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/testchainca.cert with env {}
W20260927 22:53:34.872866 14267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:34.873138 14267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:34.873221 14267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:34.880309 14267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:34.880404 14267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:34.880441 14267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:34.880489 14267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:34.880532 14267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:53:34.880602 14267 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:53:34.880642 14267 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37259/jwks.json
I20260927 22:53:34.889891 14267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:45197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37259/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:34.890995 14267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:34.892151 14267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:34.903506 14272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:34.903456 14275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:34.903434 14273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:34.904011 14267 server_base.cc:1034] running on GCE node
I20260927 22:53:34.904742 14267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:34.905993 14267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:34.907196 14267 hybrid_clock.cc:648] HybridClock initialized: now 1790549614907161 us; error 35 us; skew 500 ppm
I20260927 22:53:34.907557 14267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:34.909574 14267 webserver.cc:460] Webserver started at http://127.8.204.190:37483/ using document root <none> and password file <none>
I20260927 22:53:34.910194 14267 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:34.910280 14267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:34.910544 14267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:34.913288 14267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "66d309c575904afe8d4b04c62f0f3b03"
format_stamp: "Formatted at 2026-09-27 22:53:34 on dist-test-slave-40vx"
server_key: "bcaced43ee4728edb752041afc44d603"
server_key_iv: "eed8bccc9bbca3f3964d19c62f9c2d50"
server_key_version: "encryptionkey@0"
I20260927 22:53:34.914000 14267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "66d309c575904afe8d4b04c62f0f3b03"
format_stamp: "Formatted at 2026-09-27 22:53:34 on dist-test-slave-40vx"
server_key: "bcaced43ee4728edb752041afc44d603"
server_key_iv: "eed8bccc9bbca3f3964d19c62f9c2d50"
server_key_version: "encryptionkey@0"
I20260927 22:53:34.918848 14267 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 22:53:34.922065 14281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:34.923359 14267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:34.923527 14267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "66d309c575904afe8d4b04c62f0f3b03"
format_stamp: "Formatted at 2026-09-27 22:53:34 on dist-test-slave-40vx"
server_key: "bcaced43ee4728edb752041afc44d603"
server_key_iv: "eed8bccc9bbca3f3964d19c62f9c2d50"
server_key_version: "encryptionkey@0"
I20260927 22:53:34.923719 14267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:34.944012 14267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:34.944814 14267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:34.945116 14267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:34.962062 14267 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:45197
I20260927 22:53:34.962081 14332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:45197 every 8 connection(s)
I20260927 22:53:34.963404 14267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:34.965720 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14267
I20260927 22:53:34.965925 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:34.966640 9010 external_mini_cluster.cc:1428] Setting key 9686c769c46d02c79d782e30d66efc29
I20260927 22:53:34.967243 14333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:34.981909 14333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03: Bootstrap starting.
I20260927 22:53:34.981909 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:34.968606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.8.204.190:45197)
0927 22:53:34.969627 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:34.969641 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:34.969688 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:53:34.971981 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:34.972007 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:34.972603 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:34.973023 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:34.973535 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.975512 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.976922 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:34.977944 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:34.978127 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:34.979186 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:34.979195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:34.979212 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:34.979963 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:34.980048 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:34.980071 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:34.980175 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:34.980388 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 22:53:34.980696 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":333,"threads_started":1}
I20260927 22:53:34.985395 14333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:34.985642 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:53:34.986425 14333 log.cc:824] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:34.989142 14333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03: No bootstrap required, opened a new log
I20260927 22:53:34.996428 14333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d309c575904afe8d4b04c62f0f3b03" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45197 } }
I20260927 22:53:34.996798 14333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:34.996906 14333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d309c575904afe8d4b04c62f0f3b03, State: Initialized, Role: FOLLOWER
I20260927 22:53:34.997550 14333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d309c575904afe8d4b04c62f0f3b03" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45197 } }
I20260927 22:53:34.997813 14333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:34.997952 14333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:34.998129 14333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:35.000322 14333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d309c575904afe8d4b04c62f0f3b03" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45197 } }
I20260927 22:53:35.000845 14333 leader_election.cc:302] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66d309c575904afe8d4b04c62f0f3b03; no voters:
I20260927 22:53:35.001420 14333 leader_election.cc:288] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:35.001591 14338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:35.002799 14338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [term 1 LEADER]: Becoming Leader. State: Replica: 66d309c575904afe8d4b04c62f0f3b03, State: Running, Role: LEADER
I20260927 22:53:35.003391 14338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d309c575904afe8d4b04c62f0f3b03" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45197 } }
I20260927 22:53:35.003683 14333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:35.006247 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66d309c575904afe8d4b04c62f0f3b03. Latest consensus state: current_term: 1 leader_uuid: "66d309c575904afe8d4b04c62f0f3b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d309c575904afe8d4b04c62f0f3b03" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45197 } } }
I20260927 22:53:35.006714 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:35.006361 14339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66d309c575904afe8d4b04c62f0f3b03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66d309c575904afe8d4b04c62f0f3b03" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45197 } } }
I20260927 22:53:35.007431 14339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:35.009011 14347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:35.013374 14347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:35.021083 14347 catalog_manager.cc:1347] Generated new cluster ID: b286d5b14f094e91879927a5ae409f3f
I20260927 22:53:35.021199 14347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:35.045996 14347 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:35.047330 14347 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:35.057906 14347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66d309c575904afe8d4b04c62f0f3b03: Generated new TSK 0
I20260927 22:53:35.058701 14347 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:53:35.091352 14363 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:53:35.091482 14363 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:53:35.092640 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:35.073694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.8.204.190:45197)
0927 22:53:35.073843 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:35.073848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:35.074334 (+ 486us) server_negotiation.cc:407] Connection header received
0927 22:53:35.074639 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:35.074643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:35.074717 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:35.074863 (+ 146us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:53:35.075361 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.076069 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.077478 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.077921 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.078056 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:35.078531 (+ 475us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:53:35.091952 (+ 13421us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:53:35.092318 (+ 366us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 22:53:35.092758 14362 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:35.073700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45197 (local address 127.0.0.1:60804)
0927 22:53:35.074229 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:35.074244 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:35.074456 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:35.074877 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:35.074882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:35.074894 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:53:35.075161 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:35.075170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.076202 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.076210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:35.077287 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:35.077297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.078105 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.078114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:35.078261 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:35.078297 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:53:35.092148 (+ 13851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:53:35.092372 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.190:45197: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":294,"threads_started":1}
W20260927 22:53:35.092762 14336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60804: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37259/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 22:53:35.094098 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14267
I20260927 22:53:35.095408 14328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:35.187119 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14267
2026-09-27T22:53:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (709 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:53:35.241751 9010 mini_oidc.cc:152] Starting JWKS server
I20260927 22:53:35.243574 9010 webserver.cc:460] Webserver started at https://127.0.0.1:36025/ using document root <none> and password file <none>
I20260927 22:53:35.243827 9010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:53:35.244417 9010 webserver.cc:460] Webserver started at http://127.0.0.1:36303/ using document root <none> and password file <none>
2026-09-27T22:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:35Z Disabled control of system clock
I20260927 22:53:35.270277 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45523
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:39005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:45523
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36025/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:53:35.581152 14370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:35.581526 14370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:35.581682 14370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:35.588810 14370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:35.588954 14370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:35.589048 14370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:35.589147 14370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:35.589234 14370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:53:35.589322 14370 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:53:35.589380 14370 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36025/jwks.json
I20260927 22:53:35.598264 14370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:45523
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36025/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:35.599599 14370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:35.601372 14370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:35.612874 14376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:35.612900 14378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:35.612896 14375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:35.614956 14370 server_base.cc:1034] running on GCE node
I20260927 22:53:35.615792 14370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:35.617174 14370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:35.618377 14370 hybrid_clock.cc:648] HybridClock initialized: now 1790549615618332 us; error 38 us; skew 500 ppm
I20260927 22:53:35.618717 14370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:35.620956 14370 webserver.cc:460] Webserver started at http://127.8.204.190:39187/ using document root <none> and password file <none>
I20260927 22:53:35.621618 14370 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:35.621730 14370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:35.622057 14370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:35.626631 14370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "c9469beaa70b45d293d6aeb778768095"
format_stamp: "Formatted at 2026-09-27 22:53:35 on dist-test-slave-40vx"
server_key: "d37a79d4bac986f8e3dc5af33b70b2ae"
server_key_iv: "742c37fea45d0919eea3dd7789ee0079"
server_key_version: "encryptionkey@0"
I20260927 22:53:35.627449 14370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "c9469beaa70b45d293d6aeb778768095"
format_stamp: "Formatted at 2026-09-27 22:53:35 on dist-test-slave-40vx"
server_key: "d37a79d4bac986f8e3dc5af33b70b2ae"
server_key_iv: "742c37fea45d0919eea3dd7789ee0079"
server_key_version: "encryptionkey@0"
I20260927 22:53:35.632256 14370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:53:35.635670 14384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:35.637043 14370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:53:35.637212 14370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "c9469beaa70b45d293d6aeb778768095"
format_stamp: "Formatted at 2026-09-27 22:53:35 on dist-test-slave-40vx"
server_key: "d37a79d4bac986f8e3dc5af33b70b2ae"
server_key_iv: "742c37fea45d0919eea3dd7789ee0079"
server_key_version: "encryptionkey@0"
I20260927 22:53:35.637424 14370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:35.651612 14370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:35.652457 14370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:35.652760 14370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:35.671288 14370 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:45523
I20260927 22:53:35.671366 14435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:45523 every 8 connection(s)
I20260927 22:53:35.672649 14370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:35.676393 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:35.677510 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14370
I20260927 22:53:35.677737 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:35.678437 9010 external_mini_cluster.cc:1428] Setting key f95053fe90e3acd2c9f670d9115a9884
I20260927 22:53:35.687832 14436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095: Bootstrap starting.
I20260927 22:53:35.691188 14436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:35.692047 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:35.680437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.8.204.190:45523)
0927 22:53:35.681462 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:35.681472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:35.681516 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:53:35.683089 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:35.683109 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:35.683615 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:35.683923 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:35.684521 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.685816 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.687620 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.688333 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.688466 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:35.689650 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:35.689657 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:35.689669 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:35.690176 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:35.690257 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:35.690277 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:35.690367 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:35.690640 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:53:35.690982 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":400,"threads_started":1}
I20260927 22:53:35.692343 14436 log.cc:824] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:35.695591 14436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095: No bootstrap required, opened a new log
I20260927 22:53:35.696012 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:53:35.702790 14436 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9469beaa70b45d293d6aeb778768095" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45523 } }
I20260927 22:53:35.703243 14436 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:35.703367 14436 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9469beaa70b45d293d6aeb778768095, State: Initialized, Role: FOLLOWER
I20260927 22:53:35.703994 14436 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9469beaa70b45d293d6aeb778768095" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45523 } }
I20260927 22:53:35.704210 14436 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:35.704331 14436 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:35.704497 14436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:35.706547 14436 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9469beaa70b45d293d6aeb778768095" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45523 } }
I20260927 22:53:35.707095 14436 leader_election.cc:302] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9469beaa70b45d293d6aeb778768095; no voters:
I20260927 22:53:35.707724 14436 leader_election.cc:288] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:35.707894 14441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:35.708588 14441 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [term 1 LEADER]: Becoming Leader. State: Replica: c9469beaa70b45d293d6aeb778768095, State: Running, Role: LEADER
I20260927 22:53:35.709098 14441 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9469beaa70b45d293d6aeb778768095" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45523 } }
I20260927 22:53:35.709746 14436 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:35.712076 14443 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9469beaa70b45d293d6aeb778768095. Latest consensus state: current_term: 1 leader_uuid: "c9469beaa70b45d293d6aeb778768095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9469beaa70b45d293d6aeb778768095" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45523 } } }
I20260927 22:53:35.712499 14443 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:35.716318 14450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:35.716284 14442 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9469beaa70b45d293d6aeb778768095" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9469beaa70b45d293d6aeb778768095" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45523 } } }
I20260927 22:53:35.716706 14442 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:35.720919 14450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:35.729787 14450 catalog_manager.cc:1347] Generated new cluster ID: f494e0b7d38c46d29df3a3de336c059b
I20260927 22:53:35.729892 14450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:35.739779 14450 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:35.741108 14450 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:35.750967 14450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9469beaa70b45d293d6aeb778768095: Generated new TSK 0
I20260927 22:53:35.751984 14450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:53:35.779219 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:35.770836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.8.204.190:45523)
0927 22:53:35.771041 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:35.771048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:35.771483 (+ 435us) server_negotiation.cc:407] Connection header received
0927 22:53:35.771686 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:35.771692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:35.771802 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:35.771974 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:53:35.772394 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.773518 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.774880 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.775579 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.775711 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:35.776231 (+ 520us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:53:35.778581 (+ 2350us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:53:35.778910 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 22:53:35.779369 14439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43554: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 22:53:35.781232 14465 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:35.770708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45523 (local address 127.0.0.1:43554)
0927 22:53:35.771383 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:35.771400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:35.771515 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:35.771991 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:35.771994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:35.772006 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:53:35.772254 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:35.772262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.773664 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.773667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:35.774742 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:35.774754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:35.775775 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:35.775780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:35.775924 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:35.775958 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:53:35.778786 (+ 2828us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:53:35.780806 (+ 2020us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.190:45523: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36025/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":1729,"thread_start_us":376,"threads_started":1}
I20260927 22:53:35.782972 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14370
I20260927 22:53:35.784135 14431 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:53:35.795189 14466 webserver.cc:533] Webserver: sslize failed:
E20260927 22:53:35.796783 14466 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:53:35.880630 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14370
2026-09-27T22:53:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (689 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:53:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:35Z Disabled control of system clock
I20260927 22:53:35.955615 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42909
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:42909
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/testchainca.cert with env {}
W20260927 22:53:36.269330 14471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:36.269608 14471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:36.269757 14471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:36.276911 14471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:36.277011 14471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:36.277048 14471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:36.277122 14471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:36.277185 14471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:53:36.277310 14471 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:53:36.277354 14471 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:53:36.287201 14471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:42909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:36.288504 14471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:36.289768 14471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:36.301491 14477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:36.301501 14476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:36.301959 14479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:36.302414 14471 server_base.cc:1034] running on GCE node
I20260927 22:53:36.303131 14471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:36.304456 14471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:36.305641 14471 hybrid_clock.cc:648] HybridClock initialized: now 1790549616305607 us; error 29 us; skew 500 ppm
I20260927 22:53:36.305986 14471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:53:36.308163 14471 webserver.cc:460] Webserver started at http://127.8.204.190:46587/ using document root <none> and password file <none>
I20260927 22:53:36.308822 14471 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:36.308933 14471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:36.309247 14471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:36.311977 14471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "36ed52ee633346e9b1466f2bb589ebcc"
format_stamp: "Formatted at 2026-09-27 22:53:36 on dist-test-slave-40vx"
server_key: "863f4675f01a8861571ec5fa5d2c2fcc"
server_key_iv: "cc452cf6d39bc6018145b8c85b31a575"
server_key_version: "encryptionkey@0"
I20260927 22:53:36.312754 14471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "36ed52ee633346e9b1466f2bb589ebcc"
format_stamp: "Formatted at 2026-09-27 22:53:36 on dist-test-slave-40vx"
server_key: "863f4675f01a8861571ec5fa5d2c2fcc"
server_key_iv: "cc452cf6d39bc6018145b8c85b31a575"
server_key_version: "encryptionkey@0"
I20260927 22:53:36.317484 14471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 22:53:36.320751 14485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:36.322140 14471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:53:36.322315 14471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "36ed52ee633346e9b1466f2bb589ebcc"
format_stamp: "Formatted at 2026-09-27 22:53:36 on dist-test-slave-40vx"
server_key: "863f4675f01a8861571ec5fa5d2c2fcc"
server_key_iv: "cc452cf6d39bc6018145b8c85b31a575"
server_key_version: "encryptionkey@0"
I20260927 22:53:36.322525 14471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:36.342397 14471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:36.343232 14471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:36.343564 14471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:36.361352 14471 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:42909
I20260927 22:53:36.361373 14536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:42909 every 8 connection(s)
I20260927 22:53:36.362619 14471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:36.366291 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:36.372592 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14471
I20260927 22:53:36.372784 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:36.373585 9010 external_mini_cluster.cc:1428] Setting key ac156c5fda30a24b7d34efd0770605e6
I20260927 22:53:36.377326 14537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc: Bootstrap starting.
I20260927 22:53:36.380410 14537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:36.381381 14537 log.cc:824] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:36.384497 14537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc: No bootstrap required, opened a new log
I20260927 22:53:36.387056 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:36.375488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.8.204.190:42909)
0927 22:53:36.376361 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:36.376371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:36.376408 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:53:36.377989 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:36.378009 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:36.378431 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:36.378803 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:36.379347 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:36.380716 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:36.382176 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:36.383100 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:36.383284 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:36.384382 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:36.384392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:36.384409 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:53:36.385113 (+ 704us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:36.385212 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:53:36.385250 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:36.385374 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:36.385517 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:53:36.385814 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":289,"threads_started":1}
I20260927 22:53:36.390653 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:53:36.393285 14537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed52ee633346e9b1466f2bb589ebcc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42909 } }
I20260927 22:53:36.393719 14537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:36.393818 14537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ed52ee633346e9b1466f2bb589ebcc, State: Initialized, Role: FOLLOWER
I20260927 22:53:36.394368 14537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed52ee633346e9b1466f2bb589ebcc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42909 } }
I20260927 22:53:36.394591 14537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:36.394737 14537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:36.394908 14537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:36.397351 14537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed52ee633346e9b1466f2bb589ebcc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42909 } }
I20260927 22:53:36.397832 14537 leader_election.cc:302] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36ed52ee633346e9b1466f2bb589ebcc; no voters:
I20260927 22:53:36.398404 14537 leader_election.cc:288] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:36.398597 14542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:36.399425 14542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [term 1 LEADER]: Becoming Leader. State: Replica: 36ed52ee633346e9b1466f2bb589ebcc, State: Running, Role: LEADER
I20260927 22:53:36.400039 14542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed52ee633346e9b1466f2bb589ebcc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42909 } }
I20260927 22:53:36.400550 14537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:36.402711 14543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36ed52ee633346e9b1466f2bb589ebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed52ee633346e9b1466f2bb589ebcc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42909 } } }
I20260927 22:53:36.403062 14543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:36.403434 14544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36ed52ee633346e9b1466f2bb589ebcc. Latest consensus state: current_term: 1 leader_uuid: "36ed52ee633346e9b1466f2bb589ebcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36ed52ee633346e9b1466f2bb589ebcc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42909 } } }
I20260927 22:53:36.404664 14544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:36.407066 14550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:36.412031 14550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:36.420279 14550 catalog_manager.cc:1347] Generated new cluster ID: b7c0125e2e074765b40c51b547f2354d
I20260927 22:53:36.420392 14550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:36.430634 14550 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:36.432389 14550 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:36.440157 14550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36ed52ee633346e9b1466f2bb589ebcc: Generated new TSK 0
I20260927 22:53:36.441102 14550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:53:36.639518 14540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:36.466383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.8.204.190:42909)
0927 22:53:36.466591 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:36.466598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:36.466904 (+ 306us) server_negotiation.cc:407] Connection header received
0927 22:53:36.467132 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:36.467138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:36.467222 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:36.467405 (+ 183us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:53:36.467878 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:36.469010 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:36.470370 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:36.471059 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:36.471186 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:36.471793 (+ 607us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:53:36.638708 (+166915us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:53:36.639165 (+ 457us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 22:53:36.639655 14540 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:53:36.639730 14566 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:36.466205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42909 (local address 127.0.0.1:44802)
0927 22:53:36.466802 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:36.466818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:36.466948 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:36.467452 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:36.467457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:36.467470 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:53:36.467723 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:36.467731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:36.469164 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:36.469167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:36.470229 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:36.470239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:36.471243 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:36.471248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:36.471460 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:36.471498 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:53:36.639055 (+167557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:53:36.639359 (+ 304us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.204.190:42909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":355,"threads_started":1}
I20260927 22:53:36.641793 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14471
I20260927 22:53:36.642946 14532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:36.748492 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14471
2026-09-27T22:53:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (853 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (398 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (395 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:37 dist-test-slave-40vx krb5kdc[14581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:37 dist-test-slave-40vx krb5kdc[14581](info): set up 2 sockets
Sep 27 22:53:37 dist-test-slave-40vx krb5kdc[14581](info): commencing operation
krb5kdc: starting...
W20260927 22:53:38.627847 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:38 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:38.827554 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37209
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:39.140983 14594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:39.141297 14594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:39.141465 14594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:39.148496 14594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:39.148633 14594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:39.148737 14594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:39.148838 14594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:39.148927 14594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:39.158501 14594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:39.159781 14594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:39.161103 14594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:39.172415 14599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:39.172581 14602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:39.172673 14600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:39.173561 14594 server_base.cc:1034] running on GCE node
I20260927 22:53:39.174252 14594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:39.175621 14594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:39.176819 14594 hybrid_clock.cc:648] HybridClock initialized: now 1790549619176748 us; error 56 us; skew 500 ppm
I20260927 22:53:39.177145 14594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:39 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549619, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:39.180146 14594 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:53:39.181507 14594 webserver.cc:460] Webserver started at http://127.8.204.190:45775/ using document root <none> and password file <none>
I20260927 22:53:39.182116 14594 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:39.182212 14594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:39.182451 14594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:39.185297 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "5e8fba6b333b4d008f7d58eea17732e6"
format_stamp: "Formatted at 2026-09-27 22:53:39 on dist-test-slave-40vx"
server_key: "c99766c33fb46c1449ab2705296d306c"
server_key_iv: "b41e3156f36d5fd7ea4c363cb3c160d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:39.185988 14594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "5e8fba6b333b4d008f7d58eea17732e6"
format_stamp: "Formatted at 2026-09-27 22:53:39 on dist-test-slave-40vx"
server_key: "c99766c33fb46c1449ab2705296d306c"
server_key_iv: "b41e3156f36d5fd7ea4c363cb3c160d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:39.190924 14594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:53:39.194124 14609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:39.195412 14594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:53:39.195586 14594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "5e8fba6b333b4d008f7d58eea17732e6"
format_stamp: "Formatted at 2026-09-27 22:53:39 on dist-test-slave-40vx"
server_key: "c99766c33fb46c1449ab2705296d306c"
server_key_iv: "b41e3156f36d5fd7ea4c363cb3c160d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:39.195767 14594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:39.220746 14594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:39.224618 14594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:39.224898 14594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:39.242277 14594 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:37209
I20260927 22:53:39.242305 14660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:37209 every 8 connection(s)
I20260927 22:53:39.243637 14594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:39.246528 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14594
I20260927 22:53:39.246731 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:39.247532 9010 external_mini_cluster.cc:1428] Setting key e3bd4ce9159e463e63810d2f03471a46
I20260927 22:53:39.247752 14661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:39.259397 14661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6: Bootstrap starting.
Sep 27 22:53:39 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:39.262275 14661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:39.263412 14661 log.cc:824] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:39.266038 14661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6: No bootstrap required, opened a new log
I20260927 22:53:39.268509 14664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:39.249497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.8.204.190:37209)
0927 22:53:39.250216 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:39.250228 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:39.250267 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:53:39.252240 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:39.252260 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:39.252641 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:39.253058 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:39.254378 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:39.255676 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:39.256870 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:39.257546 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:39.257762 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:39.261629 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:39.261636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:39.261649 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:39.261681 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:39.264688 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:39.265331 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:39.265339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:39.265354 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:39.265508 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:39.265883 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:39.265888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:39.265891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:39.266402 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:39.266751 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:39.267138 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 22:53:39.267541 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":309,"threads_started":1}
I20260927 22:53:39.274528 14661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8fba6b333b4d008f7d58eea17732e6" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37209 } }
I20260927 22:53:39.274866 14661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:39.274981 14661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8fba6b333b4d008f7d58eea17732e6, State: Initialized, Role: FOLLOWER
I20260927 22:53:39.275583 14661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8fba6b333b4d008f7d58eea17732e6" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37209 } }
I20260927 22:53:39.275804 14661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:39.275949 14661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:39.276116 14661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:39.278069 14661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8fba6b333b4d008f7d58eea17732e6" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37209 } }
I20260927 22:53:39.278523 14661 leader_election.cc:302] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e8fba6b333b4d008f7d58eea17732e6; no voters:
I20260927 22:53:39.279037 14661 leader_election.cc:288] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:39.279193 14666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:39.280164 14666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [term 1 LEADER]: Becoming Leader. State: Replica: 5e8fba6b333b4d008f7d58eea17732e6, State: Running, Role: LEADER
I20260927 22:53:39.280730 14666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8fba6b333b4d008f7d58eea17732e6" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37209 } }
I20260927 22:53:39.281103 14661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:39.283344 14667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e8fba6b333b4d008f7d58eea17732e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8fba6b333b4d008f7d58eea17732e6" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37209 } } }
I20260927 22:53:39.283715 14667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:39.284039 14668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e8fba6b333b4d008f7d58eea17732e6. Latest consensus state: current_term: 1 leader_uuid: "5e8fba6b333b4d008f7d58eea17732e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e8fba6b333b4d008f7d58eea17732e6" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37209 } } }
I20260927 22:53:39.284320 14668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:39.290413 14675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:39.295382 14675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:39.304852 14675 catalog_manager.cc:1347] Generated new cluster ID: f6121f75476f4be4b8d701d2a20c9920
I20260927 22:53:39.304975 14675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:39.326287 14675 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:39.327822 14675 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:39.333714 14675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e8fba6b333b4d008f7d58eea17732e6: Generated new TSK 0
I20260927 22:53:39.334522 14675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:53:39.387071 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37209
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:39.701508 14688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:39.701778 14688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:39.701964 14688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:39.709254 14688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:39.709403 14688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:39.709594 14688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:39.719406 14688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:39.720710 14688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:39.722028 14688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:39.735598 14696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:39.735647 14693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:39.735598 14694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:39.736459 14688 server_base.cc:1034] running on GCE node
I20260927 22:53:39.737089 14688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:39.738036 14688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:39.739301 14688 hybrid_clock.cc:648] HybridClock initialized: now 1790549619739221 us; error 39 us; skew 500 ppm
I20260927 22:53:39.739636 14688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:39 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549619, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:39.742648 14688 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:53:39.744192 14688 webserver.cc:460] Webserver started at http://127.8.204.129:45491/ using document root <none> and password file <none>
I20260927 22:53:39.744813 14688 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:39.744910 14688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:39.745169 14688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:39.748040 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "ecbacfe8c7e941a6b3a16f2cffdd6d00"
format_stamp: "Formatted at 2026-09-27 22:53:39 on dist-test-slave-40vx"
server_key: "c3cf0370175cb56e7634244347095649"
server_key_iv: "1cf2ff494145e0b26038443905ddd92f"
server_key_version: "encryptionkey@0"
I20260927 22:53:39.748745 14688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "ecbacfe8c7e941a6b3a16f2cffdd6d00"
format_stamp: "Formatted at 2026-09-27 22:53:39 on dist-test-slave-40vx"
server_key: "c3cf0370175cb56e7634244347095649"
server_key_iv: "1cf2ff494145e0b26038443905ddd92f"
server_key_version: "encryptionkey@0"
I20260927 22:53:39.753587 14688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260927 22:53:39.756892 14703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:39.758194 14688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:53:39.758347 14688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "ecbacfe8c7e941a6b3a16f2cffdd6d00"
format_stamp: "Formatted at 2026-09-27 22:53:39 on dist-test-slave-40vx"
server_key: "c3cf0370175cb56e7634244347095649"
server_key_iv: "1cf2ff494145e0b26038443905ddd92f"
server_key_version: "encryptionkey@0"
I20260927 22:53:39.758548 14688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:39.775835 14688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:39.779863 14688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:39.780139 14688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:39.781111 14688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:39.782505 14688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:39.782574 14688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:39.782733 14688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:39.782815 14688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:39.817548 14688 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:35915
I20260927 22:53:39.817577 14815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:35915 every 8 connection(s)
I20260927 22:53:39.818809 14688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:39.825328 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14688
I20260927 22:53:39.825522 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:39.826179 9010 external_mini_cluster.cc:1428] Setting key e9e5295a3d769f445c1e0e696d237c63
Sep 27 22:53:39 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549619, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:39.840308 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:39.821270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:56317 (local address 127.8.204.190:37209)
0927 22:53:39.821707 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:39.821714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:39.822683 (+ 969us) server_negotiation.cc:407] Connection header received
0927 22:53:39.824432 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:39.824436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:39.824500 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:39.824601 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:39.826125 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:39.827073 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:39.828986 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:39.829366 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:39.829482 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:39.833911 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:39.833920 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:39.833936 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:39.833979 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:39.837044 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:39.837713 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:39.837722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:39.837728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:39.837871 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:39.838263 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:39.838271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:39.838277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:39.838763 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:39.838913 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:39.839586 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:53:39.839846 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":195,"threads_started":1}
I20260927 22:53:39.840924 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:39.821812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37209 (local address 127.8.204.129:56317)
0927 22:53:39.822523 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:39.822568 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:39.824193 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:39.824794 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:39.824802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:39.825303 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:39.825954 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:39.825971 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:39.827234 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:39.827238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:39.828833 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:39.828849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:39.829585 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:39.829595 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:39.829930 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:39.830672 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:39.830706 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:39.833628 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:39.837183 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:39.837189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:39.837201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:39.837576 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:39.837973 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:39.837976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:39.837978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:39.838155 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:39.838877 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:39.838882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:39.839175 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 22:53:39.839451 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 22:53:39.839824 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":291,"threads_started":1}
I20260927 22:53:39.842425 14816 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37209
I20260927 22:53:39.842727 14816 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:39.843549 14816 heartbeater.cc:511] Master 127.8.204.190:37209 requested a full tablet report, sending...
I20260927 22:53:39.845589 14626 ts_manager.cc:194] Registered new tserver with Master: ecbacfe8c7e941a6b3a16f2cffdd6d00 (127.8.204.129:35915)
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:53:39.847060 14626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:56317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:53:39.874424 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37209
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:40.209606 14822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:40.209860 14822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:40.209949 14822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:40.216805 14822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:40.216897 14822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:40.216998 14822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:40.225870 14822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:40.226964 14822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:40.228237 14822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:40.240736 14830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:40.240762 14828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:40.240818 14827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:40.241725 14822 server_base.cc:1034] running on GCE node
I20260927 22:53:40.242322 14822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:40.243198 14822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:40.244428 14822 hybrid_clock.cc:648] HybridClock initialized: now 1790549620244379 us; error 44 us; skew 500 ppm
I20260927 22:53:40.244722 14822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.247890 14822 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:53:40.249356 14822 webserver.cc:460] Webserver started at http://127.8.204.130:35479/ using document root <none> and password file <none>
I20260927 22:53:40.249979 14822 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:40.250114 14822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:40.250411 14822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:40.253247 14822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "4a0b94e056e64bbe80169c8de8fe6a4b"
format_stamp: "Formatted at 2026-09-27 22:53:40 on dist-test-slave-40vx"
server_key: "6e1d6bdcdc74fd33f3716435b86b9c9a"
server_key_iv: "5090062ac8f9caf308b091f513f1c82b"
server_key_version: "encryptionkey@0"
I20260927 22:53:40.253973 14822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "4a0b94e056e64bbe80169c8de8fe6a4b"
format_stamp: "Formatted at 2026-09-27 22:53:40 on dist-test-slave-40vx"
server_key: "6e1d6bdcdc74fd33f3716435b86b9c9a"
server_key_iv: "5090062ac8f9caf308b091f513f1c82b"
server_key_version: "encryptionkey@0"
I20260927 22:53:40.258668 14822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:53:40.261996 14837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:40.263366 14822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:53:40.263537 14822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "4a0b94e056e64bbe80169c8de8fe6a4b"
format_stamp: "Formatted at 2026-09-27 22:53:40 on dist-test-slave-40vx"
server_key: "6e1d6bdcdc74fd33f3716435b86b9c9a"
server_key_iv: "5090062ac8f9caf308b091f513f1c82b"
server_key_version: "encryptionkey@0"
I20260927 22:53:40.263777 14822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:40.275163 14822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:40.278774 14822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:40.279057 14822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:40.279980 14822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:40.281267 14822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:40.281337 14822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:40.281462 14822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:40.281509 14822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:40.313378 14822 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:44397
I20260927 22:53:40.313418 14949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:44397 every 8 connection(s)
I20260927 22:53:40.314628 14822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:40.323956 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14822
I20260927 22:53:40.324126 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:40.324565 9010 external_mini_cluster.cc:1428] Setting key 443741f6f65ed719d95b4e1f9241b6b0
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:40.333513 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.317116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:33915 (local address 127.8.204.190:37209)
0927 22:53:40.317282 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.317286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.318420 (+ 1134us) server_negotiation.cc:407] Connection header received
0927 22:53:40.320147 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.320151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.320221 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.320308 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.321668 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.322364 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.323928 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.324439 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.324564 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.328065 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:40.328070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:40.328074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:40.328101 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:40.330724 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.331410 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.331415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.331419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.331549 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.331960 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.331967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.331975 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.332220 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:40.332368 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:40.332995 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 22:53:40.333195 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:53:40.334602 14952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.317587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37209 (local address 127.8.204.130:33915)
0927 22:53:40.318256 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:40.318300 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:40.319910 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:40.320495 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:40.320502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:40.320992 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:40.321512 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:40.321526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.322493 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.322497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:40.323779 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:40.323789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.324664 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.324671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:40.324877 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.325531 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:40.325560 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:40.327864 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:40.330854 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:40.330862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:40.330876 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:40.331266 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:40.331662 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:40.331668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:40.331671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:40.331843 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:40.332341 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:40.332347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:40.332667 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 22:53:40.332904 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 22:53:40.333372 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260927 22:53:40.335940 14950 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37209
I20260927 22:53:40.336231 14950 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:40.336922 14950 heartbeater.cc:511] Master 127.8.204.190:37209 requested a full tablet report, sending...
I20260927 22:53:40.338315 14626 ts_manager.cc:194] Registered new tserver with Master: 4a0b94e056e64bbe80169c8de8fe6a4b (127.8.204.130:44397)
I20260927 22:53:40.339571 14626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:33915
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:53:40.371068 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37209
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:40.692674 14955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:40.693069 14955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:40.693238 14955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:40.702028 14955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:40.702136 14955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:40.702337 14955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:40.713256 14955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:40.714614 14955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:40.715986 14955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:40.729178 14960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:40.729110 14963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:40.729139 14961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:40.730152 14955 server_base.cc:1034] running on GCE node
I20260927 22:53:40.730816 14955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:40.731858 14955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:40.733058 14955 hybrid_clock.cc:648] HybridClock initialized: now 1790549620733018 us; error 33 us; skew 500 ppm
I20260927 22:53:40.733383 14955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.736351 14955 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:53:40.737748 14955 webserver.cc:460] Webserver started at http://127.8.204.131:41051/ using document root <none> and password file <none>
I20260927 22:53:40.738384 14955 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:40.738499 14955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:40.738819 14955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:40.741645 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "9b0552e383f54b2eb51035dda0688b7c"
format_stamp: "Formatted at 2026-09-27 22:53:40 on dist-test-slave-40vx"
server_key: "c37326e7a48e785ffcebff5710672911"
server_key_iv: "f2c84bee97d589ad76e5331b50ab60ba"
server_key_version: "encryptionkey@0"
I20260927 22:53:40.742385 14955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "9b0552e383f54b2eb51035dda0688b7c"
format_stamp: "Formatted at 2026-09-27 22:53:40 on dist-test-slave-40vx"
server_key: "c37326e7a48e785ffcebff5710672911"
server_key_iv: "f2c84bee97d589ad76e5331b50ab60ba"
server_key_version: "encryptionkey@0"
I20260927 22:53:40.747228 14955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:53:40.750499 14970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:40.751850 14955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:40.752018 14955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "9b0552e383f54b2eb51035dda0688b7c"
format_stamp: "Formatted at 2026-09-27 22:53:40 on dist-test-slave-40vx"
server_key: "c37326e7a48e785ffcebff5710672911"
server_key_iv: "f2c84bee97d589ad76e5331b50ab60ba"
server_key_version: "encryptionkey@0"
I20260927 22:53:40.752216 14955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:40.773664 14955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:40.777555 14955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:40.777838 14955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:40.778805 14955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:40.780198 14955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:40.780304 14955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:40.780444 14955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:40.780561 14955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:40.813936 14955 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:33379
I20260927 22:53:40.814019 15082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:33379 every 8 connection(s)
I20260927 22:53:40.815156 14955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:40.819619 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 14955
I20260927 22:53:40.819795 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:40.820362 9010 external_mini_cluster.cc:1428] Setting key e9590ccd8ea45275d6c1d57d3a4d033b
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:53:40.835067 14818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.817803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:55639 (local address 127.8.204.190:37209)
0927 22:53:40.817982 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.817987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.819392 (+ 1405us) server_negotiation.cc:407] Connection header received
0927 22:53:40.821843 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.821849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.821965 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.822077 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.823938 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.824667 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.826149 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.826544 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.826648 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.829820 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:40.829824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:40.829828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:40.829857 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:40.832202 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.832875 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.832878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.832881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.832986 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.833325 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.833328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.833330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.833581 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:40.833730 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:40.834511 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 22:53:40.834745 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:53:40.836292 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.818372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37209 (local address 127.8.204.131:55639)
0927 22:53:40.819161 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:40.819243 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:40.821545 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:40.822352 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:40.822361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:40.823078 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:40.823767 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:40.823784 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.824806 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.824810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:40.826024 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:40.826035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.826734 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.826739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:40.826923 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.827588 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:40.827610 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:40.829652 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:40.832318 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:40.832324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:40.832335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:40.832770 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:40.833070 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:40.833073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:40.833075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:40.833233 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:40.833718 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:40.833723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:40.834087 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 22:53:40.834374 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 22:53:40.834851 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":332,"threads_started":1}
I20260927 22:53:40.837604 15083 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37209
I20260927 22:53:40.837903 15083 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:40.838645 15083 heartbeater.cc:511] Master 127.8.204.190:37209 requested a full tablet report, sending...
I20260927 22:53:40.839993 14626 ts_manager.cc:194] Registered new tserver with Master: 9b0552e383f54b2eb51035dda0688b7c (127.8.204.131:33379)
I20260927 22:53:40.840740 14626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:55639
I20260927 22:53:40.847870 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:53:40.849880 14816 heartbeater.cc:503] Master 127.8.204.190:37209 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/test-admin.keytab.
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.873049 9010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:53:40.877862 15090 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:40.875487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.8.204.129:35915)
0927 22:53:40.876108 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.876114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.876137 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:53:40.876193 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.876199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.876399 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.876559 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.877426 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44010: BlockingRecv error: recv got EOF from 127.0.0.1:44010 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":297,"threads_started":1}
W20260927 22:53:40.877923 9010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.879453 9010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.880800 9010 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:40.896090 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.882289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.8.204.129:35915)
0927 22:53:40.882408 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.882412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.882774 (+ 362us) server_negotiation.cc:407] Connection header received
0927 22:53:40.882904 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.882907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.882957 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.883034 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.884139 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.885036 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.886255 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.886790 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.886881 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.890654 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:40.890663 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:40.890677 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:40.890713 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:40.893511 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.894111 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.894119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.894121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.894245 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.894590 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.894594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.894596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.894956 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:40.895176 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:40.895544 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 22:53:40.895839 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260927 22:53:40.901587 15090 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:40.899977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.8.204.129:35915)
0927 22:53:40.900189 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.900195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.900461 (+ 266us) server_negotiation.cc:407] Connection header received
0927 22:53:40.900540 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.900543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.900610 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.900687 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.901357 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44022: BlockingRecv error: recv got EOF from 127.0.0.1:44022 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:53:40.901738 9010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.903158 9010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.904457 9010 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:40.917799 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.905783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.8.204.129:35915)
0927 22:53:40.905993 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.905999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.906201 (+ 202us) server_negotiation.cc:407] Connection header received
0927 22:53:40.906352 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.906356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.906421 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.906538 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.907434 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.908183 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.909320 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.909755 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.909859 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.913190 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:40.913194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:40.913198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:40.913227 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:40.915729 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.916289 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.916293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.916295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.916401 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.916686 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.916689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.916698 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.916930 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:40.917112 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:40.917322 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:53:40.917523 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 22:53:40.921947 15090 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:40.920274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.8.204.129:35915)
0927 22:53:40.920477 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.920484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.920718 (+ 234us) server_negotiation.cc:407] Connection header received
0927 22:53:40.920891 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.920896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.920971 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.921065 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.921731 (+ 666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44048: BlockingRecv error: recv got EOF from 127.0.0.1:44048 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 22:53:40.922338 9010 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790549572925352-9010-0/krb5kdc/krb5cc)
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.923836 9010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:40.925107 9010 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:53:40 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:53:40.938445 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.926312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44050 (local address 127.8.204.129:35915)
0927 22:53:40.926535 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.926541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.926735 (+ 194us) server_negotiation.cc:407] Connection header received
0927 22:53:40.926852 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.926855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.926909 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.927006 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.927982 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.928673 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.929799 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.930219 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.930305 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.933551 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:40.933555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:40.933558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:40.933587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:40.936022 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.936624 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.936632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.936635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.936765 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.937128 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.937133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.937137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.937383 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:40.937687 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:40.937968 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:53:40.938163 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:53:40.939919 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14688
I20260927 22:53:40.951176 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:40.940541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44066 (local address 127.8.204.129:35915)
0927 22:53:40.940715 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:40.940723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:40.940962 (+ 239us) server_negotiation.cc:407] Connection header received
0927 22:53:40.941139 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:40.941144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:40.941208 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:40.941303 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:40.942133 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.942803 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.944030 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:40.944428 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:40.944559 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:40.946458 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:40.946464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:40.946469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:40.946513 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:40.948786 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.949366 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.949372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.949376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.949497 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:40.949867 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:40.949874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:40.949877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:40.950122 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:40.950380 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:40.950682 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:53:40.950869 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:53:40.951740 14811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:41.034142 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14688
I20260927 22:53:41.050012 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14822
Sep 27 22:53:41 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:53:41.064870 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:41.050906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.8.204.130:44397)
0927 22:53:41.051467 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:41.051474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:41.051498 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:41.051566 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:41.051574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:41.051777 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:41.051938 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:41.052866 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:41.053876 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:41.055103 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:41.055859 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:41.055996 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:41.059647 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:41.059658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:41.059673 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:41.059706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:41.062449 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:41.063048 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:41.063054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:41.063056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:41.063179 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:41.063577 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:41.063580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:41.063582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:41.063894 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:41.064112 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:41.064375 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 22:53:41.064595 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
I20260927 22:53:41.066085 14945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:41.146629 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14822
I20260927 22:53:41.162235 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14955
Sep 27 22:53:41 dist-test-slave-40vx krb5kdc[14581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:53:41.177564 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:41.163137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.8.204.131:33379)
0927 22:53:41.163843 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:41.163850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:41.163881 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:53:41.163949 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:41.163955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:41.164162 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:41.164292 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:41.165202 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:41.166278 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:41.167528 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:41.168066 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:41.168160 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:41.171904 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:41.171915 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:41.171930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:41.171961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:41.174832 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:41.175673 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:41.175680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:41.175682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:41.175797 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:41.176146 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:41.176150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:41.176152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:41.176512 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:41.176734 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:41.176966 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 22:53:41.177228 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":412,"threads_started":1}
I20260927 22:53:41.178773 15078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:41.261243 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14955
I20260927 22:53:41.276625 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 14594
I20260927 22:53:41.277752 14656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:41.379783 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 14594
2026-09-27T22:53:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3840 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:41 dist-test-slave-40vx krb5kdc[15110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:41 dist-test-slave-40vx krb5kdc[15110](info): set up 2 sockets
Sep 27 22:53:41 dist-test-slave-40vx krb5kdc[15110](info): commencing operation
krb5kdc: starting...
W20260927 22:53:42.456059 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:42 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab.
Entry for principal oryx/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab.
I20260927 22:53:42.650502 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44551
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab
--principal=oryx/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:42.965036 15123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:42.965322 15123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:42.965487 15123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:42.972679 15123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:42.972785 15123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:42.972886 15123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:42.972988 15123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:42.973057 15123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:42.982657 15123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab
--principal=oryx/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:42.983959 15123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:42.985244 15123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:42.996913 15128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:42.996958 15131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:42.997011 15129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:42.997802 15123 server_base.cc:1034] running on GCE node
I20260927 22:53:42.998513 15123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:42.999912 15123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:43.001109 15123 hybrid_clock.cc:648] HybridClock initialized: now 1790549623001053 us; error 48 us; skew 500 ppm
I20260927 22:53:43.001456 15123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:43 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549623, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:43.008531 15123 init.cc:377] Logged in from keytab as oryx/127.8.204.190@KRBTEST.COM (short username oryx)
I20260927 22:53:43.009997 15123 webserver.cc:460] Webserver started at http://127.8.204.190:37979/ using document root <none> and password file <none>
I20260927 22:53:43.010694 15123 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:43.010840 15123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:43.011139 15123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:43.014266 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "36a20dcf2145413292db3c418e7e97ed"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8cadd3810aff116c98a9d1c72c37c88f"
server_key_iv: "8514f7d425ce0235f075a1efad331976"
server_key_version: "encryptionkey@0"
I20260927 22:53:43.015051 15123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "36a20dcf2145413292db3c418e7e97ed"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8cadd3810aff116c98a9d1c72c37c88f"
server_key_iv: "8514f7d425ce0235f075a1efad331976"
server_key_version: "encryptionkey@0"
I20260927 22:53:43.020274 15123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:53:43.023561 15138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:43.024821 15123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:53:43.025002 15123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "36a20dcf2145413292db3c418e7e97ed"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8cadd3810aff116c98a9d1c72c37c88f"
server_key_iv: "8514f7d425ce0235f075a1efad331976"
server_key_version: "encryptionkey@0"
I20260927 22:53:43.025187 15123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:43.047075 15123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:43.051005 15123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:43.051330 15123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:43.069567 15123 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:44551
I20260927 22:53:43.069587 15189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:44551 every 8 connection(s)
I20260927 22:53:43.070899 15123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:43.074476 15190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:43.079474 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15123
I20260927 22:53:43.079665 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:43.080444 9010 external_mini_cluster.cc:1428] Setting key a687f9ab20d53b46b283fbed061de2a5
I20260927 22:53:43.085528 15190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Bootstrap starting.
I20260927 22:53:43.089172 15190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:43.090355 15190 log.cc:824] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:43.093662 15190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: No bootstrap required, opened a new log
Sep 27 22:53:43 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:43.104069 15190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } }
I20260927 22:53:43.104480 15190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:43.104594 15190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a20dcf2145413292db3c418e7e97ed, State: Initialized, Role: FOLLOWER
I20260927 22:53:43.104801 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:43.082554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.8.204.190:44551)
0927 22:53:43.083415 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:43.083430 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:43.083469 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:53:43.085071 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:43.085093 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:43.085484 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:43.085845 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:43.087567 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:43.088887 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:43.090644 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:43.091786 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:43.091948 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:43.096512 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:43.096521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:43.096542 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:43.096591 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:43.100503 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:43.101163 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:43.101171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:43.101179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:43.101367 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:43.101721 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:43.101730 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:43.101733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:43.102412 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:43.102702 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:43.103101 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 22:53:43.103536 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":418,"threads_started":1}
I20260927 22:53:43.105226 15190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } }
I20260927 22:53:43.105456 15190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:43.105643 15190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:43.105819 15190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:43.108117 15190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } }
I20260927 22:53:43.108577 15190 leader_election.cc:302] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36a20dcf2145413292db3c418e7e97ed; no voters:
I20260927 22:53:43.109148 15190 leader_election.cc:288] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:43.109297 15195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:43.110149 15195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 LEADER]: Becoming Leader. State: Replica: 36a20dcf2145413292db3c418e7e97ed, State: Running, Role: LEADER
I20260927 22:53:43.110668 15195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } }
I20260927 22:53:43.111380 15190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:43.114033 15197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a20dcf2145413292db3c418e7e97ed. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } }
I20260927 22:53:43.114449 15197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:43.114233 15196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } }
I20260927 22:53:43.114629 15196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:43.121225 15200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:43.126509 15200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:43.135331 15200 catalog_manager.cc:1347] Generated new cluster ID: d55fb42e8da6423e8e026d2983783b7a
I20260927 22:53:43.135449 15200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:43.163501 15200 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:43.165406 15200 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:43.176712 15200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Generated new TSK 0
I20260927 22:53:43.177577 15200 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab.
Entry for principal oryx/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab.
I20260927 22:53:43.226981 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44551
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:43.559346 15216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:43.559695 15216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:43.559866 15216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:43.567651 15216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:43.567804 15216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:43.567977 15216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:43.577530 15216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:43.578851 15216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:43.580206 15216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:43.592855 15222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:43.593024 15221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:43.592960 15224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:43.593606 15216 server_base.cc:1034] running on GCE node
I20260927 22:53:43.594211 15216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:43.595110 15216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:43.596324 15216 hybrid_clock.cc:648] HybridClock initialized: now 1790549623596251 us; error 72 us; skew 500 ppm
I20260927 22:53:43.596621 15216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:43 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549623, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:43.599696 15216 init.cc:377] Logged in from keytab as oryx/127.8.204.129@KRBTEST.COM (short username oryx)
I20260927 22:53:43.601119 15216 webserver.cc:460] Webserver started at http://127.8.204.129:40123/ using document root <none> and password file <none>
I20260927 22:53:43.601794 15216 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:43.601938 15216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:43.602221 15216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:43.605023 15216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "58fd2a85f6534816b8a77be4c332c34a"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8bc3a0b2f21ad52c40cad34cf130c500"
server_key_iv: "17eecb7c840e0393ba2e3613bf3f4a79"
server_key_version: "encryptionkey@0"
I20260927 22:53:43.605774 15216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "58fd2a85f6534816b8a77be4c332c34a"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8bc3a0b2f21ad52c40cad34cf130c500"
server_key_iv: "17eecb7c840e0393ba2e3613bf3f4a79"
server_key_version: "encryptionkey@0"
I20260927 22:53:43.610720 15216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:53:43.614387 15231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:43.615818 15216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:43.615979 15216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "58fd2a85f6534816b8a77be4c332c34a"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8bc3a0b2f21ad52c40cad34cf130c500"
server_key_iv: "17eecb7c840e0393ba2e3613bf3f4a79"
server_key_version: "encryptionkey@0"
I20260927 22:53:43.616176 15216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:43.634497 15216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:43.638301 15216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:43.638564 15216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:43.639536 15216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:43.640799 15216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:43.640897 15216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:43.641036 15216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:43.641105 15216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:43.672953 15216 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41189
I20260927 22:53:43.672992 15343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41189 every 8 connection(s)
I20260927 22:53:43.674162 15216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:43.678423 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15216
I20260927 22:53:43.678632 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:43.679234 9010 external_mini_cluster.cc:1428] Setting key a1e98a98d830ff066ae0f966db1aef2a
Sep 27 22:53:43 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549623, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:43.693397 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:43.676623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:33159 (local address 127.8.204.190:44551)
0927 22:53:43.677124 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:43.677130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:43.677993 (+ 863us) server_negotiation.cc:407] Connection header received
0927 22:53:43.680092 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:43.680097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:43.680192 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:43.680304 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:43.681790 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:43.682453 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:43.683990 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:43.684397 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:43.684520 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:43.688032 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:43.688037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:43.688040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:43.688069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:43.690495 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:43.691174 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:43.691179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:43.691189 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:43.691356 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:43.691758 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:43.691763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:43.691766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:43.692027 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:43.692178 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:43.692843 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 22:53:43.693049 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":228,"threads_started":1}
I20260927 22:53:43.694288 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:43.677170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:33159)
0927 22:53:43.677837 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:43.677871 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:43.679806 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:43.680523 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:43.680531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:43.681132 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:43.681642 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:43.681658 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:43.682582 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:43.682588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:43.683849 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:43.683860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:43.684613 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:43.684620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:43.684832 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:43.685467 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:43.685492 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:43.687828 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:43.690638 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:43.690647 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:43.690663 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:43.691043 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:43.691459 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:43.691464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:43.691468 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:43.691645 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:43.692157 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:43.692165 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:43.692480 (+ 315us) client_negotiation.cc:769] Sending connection context
0927 22:53:43.692744 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 22:53:43.693240 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":243,"threads_started":1}
I20260927 22:53:43.695815 15344 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44551
I20260927 22:53:43.696127 15344 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:43.697048 15344 heartbeater.cc:511] Master 127.8.204.190:44551 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:53:43.699224 15155 ts_manager.cc:194] Registered new tserver with Master: 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.130@KRBTEST.COM" created.
I20260927 22:53:43.700850 15155 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:33159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab.
Entry for principal oryx/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab.
I20260927 22:53:43.723631 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44551
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:44.046651 15350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:44.046926 15350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:44.047101 15350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:44.054338 15350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:44.054445 15350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:44.054623 15350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:44.064244 15350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:44.065580 15350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:44.066908 15350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:44.083511 15355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:44.083549 15356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:44.086608 15358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:44.087080 15350 server_base.cc:1034] running on GCE node
I20260927 22:53:44.087829 15350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:44.088949 15350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:44.090214 15350 hybrid_clock.cc:648] HybridClock initialized: now 1790549624090153 us; error 41 us; skew 500 ppm
I20260927 22:53:44.090615 15350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:44 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549624, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:44.093999 15350 init.cc:377] Logged in from keytab as oryx/127.8.204.130@KRBTEST.COM (short username oryx)
I20260927 22:53:44.095594 15350 webserver.cc:460] Webserver started at http://127.8.204.130:32925/ using document root <none> and password file <none>
I20260927 22:53:44.096370 15350 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:44.096560 15350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:44.096882 15350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:44.100170 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "0227056099ad4354bde4a3cae8940c96"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "1f100037c82a4d2b6467c8cec84d33c8"
server_key_iv: "77b5eb254ff5b566933469561b8ce6f4"
server_key_version: "encryptionkey@0"
I20260927 22:53:44.100993 15350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "0227056099ad4354bde4a3cae8940c96"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "1f100037c82a4d2b6467c8cec84d33c8"
server_key_iv: "77b5eb254ff5b566933469561b8ce6f4"
server_key_version: "encryptionkey@0"
I20260927 22:53:44.106290 15350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 22:53:44.109694 15365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:44.110961 15350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:53:44.111153 15350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "0227056099ad4354bde4a3cae8940c96"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "1f100037c82a4d2b6467c8cec84d33c8"
server_key_iv: "77b5eb254ff5b566933469561b8ce6f4"
server_key_version: "encryptionkey@0"
I20260927 22:53:44.111382 15350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:44.135064 15350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:44.139236 15350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:44.139535 15350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:44.140390 15350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:44.141661 15350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:44.141733 15350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:44.141882 15350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:44.141935 15350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:44.174137 15350 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:42231
I20260927 22:53:44.174187 15477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:42231 every 8 connection(s)
I20260927 22:53:44.175426 15350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:44.177238 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15350
I20260927 22:53:44.177408 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:44.177852 9010 external_mini_cluster.cc:1428] Setting key 353a2a1de20067014e4de2e4e26719e2
Sep 27 22:53:44 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549624, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:44.196076 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:44.178214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:49625 (local address 127.8.204.190:44551)
0927 22:53:44.178415 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:44.178421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:44.179811 (+ 1390us) server_negotiation.cc:407] Connection header received
0927 22:53:44.181673 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:44.181678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:44.181766 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:44.181881 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:44.183353 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.184041 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.186250 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.186675 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.186810 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:44.190141 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:44.190145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:44.190149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:44.190179 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:44.192783 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:44.193786 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:44.193790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:44.193792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:44.193913 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:44.194407 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:44.194410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:44.194411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:44.194669 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:44.194784 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:44.195670 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 22:53:44.195831 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for oryx/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:53:44.197557 15480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:44.178809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:49625)
0927 22:53:44.179654 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:44.179706 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:44.181388 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:44.182112 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:44.182120 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:44.182607 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:44.183169 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:44.183186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.184190 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.184196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:44.186105 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:44.186116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.186895 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.186901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:44.187130 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:44.187840 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:44.187862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:44.189917 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:44.192948 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:44.192960 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:44.192980 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:44.193650 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:44.194027 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:44.194035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:44.194041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:44.194298 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:44.194793 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:44.194800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:44.195241 (+ 441us) client_negotiation.cc:769] Sending connection context
0927 22:53:44.195598 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 22:53:44.196139 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":326,"threads_started":1}
I20260927 22:53:44.198887 15478 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44551
I20260927 22:53:44.199232 15478 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.131@KRBTEST.COM" created.
I20260927 22:53:44.200052 15478 heartbeater.cc:511] Master 127.8.204.190:44551 requested a full tablet report, sending...
I20260927 22:53:44.201584 15155 ts_manager.cc:194] Registered new tserver with Master: 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231)
I20260927 22:53:44.202804 15155 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:49625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab.
Entry for principal oryx/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab.
I20260927 22:53:44.222632 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44551
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:44.540506 15483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:44.540827 15483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:44.540993 15483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:44.548344 15483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:44.548496 15483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:44.548663 15483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:44.558131 15483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44551
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:44.559482 15483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:44.560765 15483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:44.573764 15488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:44.573725 15489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:44.573688 15491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:44.574602 15483 server_base.cc:1034] running on GCE node
I20260927 22:53:44.575176 15483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:44.576154 15483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:44.577384 15483 hybrid_clock.cc:648] HybridClock initialized: now 1790549624577318 us; error 63 us; skew 500 ppm
I20260927 22:53:44.577716 15483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:44 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549624, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:44.580818 15483 init.cc:377] Logged in from keytab as oryx/127.8.204.131@KRBTEST.COM (short username oryx)
I20260927 22:53:44.582191 15483 webserver.cc:460] Webserver started at http://127.8.204.131:43493/ using document root <none> and password file <none>
I20260927 22:53:44.582839 15483 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:44.582983 15483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:44.583315 15483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:44.585983 15483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "ddd1d7cdcff0c1b428a3af242d0b783e"
server_key_iv: "daafc43ae143303e66b18006984b675a"
server_key_version: "encryptionkey@0"
I20260927 22:53:44.586712 15483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "ddd1d7cdcff0c1b428a3af242d0b783e"
server_key_iv: "daafc43ae143303e66b18006984b675a"
server_key_version: "encryptionkey@0"
I20260927 22:53:44.591442 15483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260927 22:53:44.594609 15498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:44.595991 15483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:44.596146 15483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "ddd1d7cdcff0c1b428a3af242d0b783e"
server_key_iv: "daafc43ae143303e66b18006984b675a"
server_key_version: "encryptionkey@0"
I20260927 22:53:44.596326 15483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:44.611492 15483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:44.615444 15483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:44.615726 15483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:44.616726 15483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:44.617964 15483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:44.618062 15483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:44.618222 15483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:44.618300 15483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:44.650316 15483 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:44645
I20260927 22:53:44.650360 15610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:44645 every 8 connection(s)
I20260927 22:53:44.651647 15483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:44.660992 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15483
I20260927 22:53:44.661201 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:44.661625 9010 external_mini_cluster.cc:1428] Setting key f7fbfde7e5daeb9e0289850e07215214
Sep 27 22:53:44 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549624, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:44.670642 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:44.654078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:44505 (local address 127.8.204.190:44551)
0927 22:53:44.654258 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:44.654263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:44.655340 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 22:53:44.657165 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:44.657171 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:44.657258 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:44.657364 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:44.658790 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.659615 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.661137 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.661603 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.661735 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:44.665347 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:44.665352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:44.665355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:44.665388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:44.667817 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:44.668515 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:44.668520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:44.668522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:44.668675 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:44.669046 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:44.669049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:44.669051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:44.669335 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:44.669445 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:44.670134 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 22:53:44.670322 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:53:44.671599 15613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:44.654481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:44505)
0927 22:53:44.655165 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:44.655204 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:44.656876 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:44.657594 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:44.657602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:44.658084 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:44.658628 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:44.658645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.659767 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.659772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:44.660994 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:44.661005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:44.661823 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:44.661832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:44.662060 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:44.662766 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:44.662796 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:44.665157 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:44.667953 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:44.667960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:44.667974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:44.668387 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:44.668775 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:44.668778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:44.668780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:44.668949 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:44.669447 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:44.669453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:44.669778 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 22:53:44.670011 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 22:53:44.670403 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":266,"threads_started":1}
I20260927 22:53:44.673108 15611 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44551
I20260927 22:53:44.673394 15611 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:44.674098 15611 heartbeater.cc:511] Master 127.8.204.190:44551 requested a full tablet report, sending...
I20260927 22:53:44.675647 15155 ts_manager.cc:194] Registered new tserver with Master: 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645)
I20260927 22:53:44.676779 15155 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:44505
I20260927 22:53:44.677218 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.204.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.189@KRBTEST.COM" created.
I20260927 22:53:44.703577 15344 heartbeater.cc:503] Master 127.8.204.190:44551 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.189.keytab.
Entry for principal oryx/127.8.204.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.189.keytab.
I20260927 22:53:44.723178 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:45951
--webserver_interface=127.8.204.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.189.keytab
--principal=oryx/127.8.204.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44551,127.8.204.189:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:45.042761 15616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:45.043066 15616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:45.043241 15616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:45.050243 15616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:45.050377 15616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:45.050493 15616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:45.050599 15616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:45.050663 15616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:45.060062 15616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44551,127.8.204.189:45951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.189.keytab
--principal=oryx/127.8.204.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
W20260927 22:53:45.060417 15616 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.204.190:44551,127.8.204.189:45951'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:53:45.061369 15616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:45.062592 15616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:45.073345 15622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:45.073436 15621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:45.074510 15624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:45.074491 15616 server_base.cc:1034] running on GCE node
I20260927 22:53:45.075389 15616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:45.076714 15616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:45.077936 15616 hybrid_clock.cc:648] HybridClock initialized: now 1790549625077860 us; error 48 us; skew 500 ppm
I20260927 22:53:45.078270 15616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:45.081380 15616 init.cc:377] Logged in from keytab as oryx/127.8.204.189@KRBTEST.COM (short username oryx)
I20260927 22:53:45.082751 15616 webserver.cc:460] Webserver started at http://127.8.204.189:33839/ using document root <none> and password file <none>
I20260927 22:53:45.083489 15616 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:45.083578 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:45.083890 15616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:45.086712 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/instance:
uuid: "d0c24fd1fec242dfaab8755016f0d8da"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "427e0b54a69e462c1ecd69f3f06d6f4f"
server_key_iv: "e43c8a42254ea57c85d46ebaf5ae1150"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.087505 15616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal/instance:
uuid: "d0c24fd1fec242dfaab8755016f0d8da"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "427e0b54a69e462c1ecd69f3f06d6f4f"
server_key_iv: "e43c8a42254ea57c85d46ebaf5ae1150"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.092684 15616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:53:45.096120 15631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:45.097463 15616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:45.097621 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
uuid: "d0c24fd1fec242dfaab8755016f0d8da"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "427e0b54a69e462c1ecd69f3f06d6f4f"
server_key_iv: "e43c8a42254ea57c85d46ebaf5ae1150"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.097834 15616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:45.117844 15616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:45.122609 15616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:45.122897 15616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:45.140311 15616 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.189:45951
I20260927 22:53:45.140326 15682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.189:45951 every 8 connection(s)
I20260927 22:53:45.141759 15616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
I20260927 22:53:45.145345 15683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:45.151986 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15616
I20260927 22:53:45.151551 15683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.189@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
I20260927 22:53:45.171602 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.154181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.8.204.190:44551)
0927 22:53:45.154379 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.154385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.155344 (+ 959us) server_negotiation.cc:407] Connection header received
0927 22:53:45.157081 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.157087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.157196 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.157298 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.158567 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.159401 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.161023 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.161548 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.161639 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.165336 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.165341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.165344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.165377 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.168388 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.169327 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.169334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.169337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.169482 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.169899 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.169910 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.169913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.170209 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.170368 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.171038 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.171244 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:53:45.172685 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.154270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41696)
0927 22:53:45.155176 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.155215 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.156832 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.157388 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.157393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.158048 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.158398 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.158414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.159523 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.159529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.160881 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.160892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.161769 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.161777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.162135 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.162899 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.162925 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.165128 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.168549 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.168560 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.168576 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.169203 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.169592 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.169598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.169602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.169779 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.170336 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.170345 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.170696 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.170979 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.171508 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":60,"spinlock_wait_cycles":17664,"thread_start_us":418,"threads_started":1}
I20260927 22:53:45.172703 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.154270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.8.204.189:45951)
0927 22:53:45.155218 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.155229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.155292 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:53:45.156811 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.156831 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.157052 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.157208 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.158648 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.160006 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.161315 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.162080 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.162202 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.166329 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.166337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.166350 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.166382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.169291 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.169930 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.169945 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.169948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.170066 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.170446 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.170452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.170455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.170812 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.171122 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.171458 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.172034 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":745,"spinlock_wait_cycles":896,"thread_start_us":525,"threads_started":1}
I20260927 22:53:45.174667 15683 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.189@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
I20260927 22:53:45.188468 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.175448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:44424)
0927 22:53:45.177574 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.177590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.177718 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.177994 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.177998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.178349 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.178467 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.178475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.179468 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.179473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.180339 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.180350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.180988 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.180993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.181176 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.181690 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.181705 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.183669 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.186156 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.186164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.186168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.186537 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.186863 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.186869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.186873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.186995 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.187535 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.187539 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.187663 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.187789 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.188064 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2050}
I20260927 22:53:45.188519 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.175402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.8.204.189:45951)
0927 22:53:45.175580 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.175587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.177664 (+ 2077us) server_negotiation.cc:407] Connection header received
0927 22:53:45.177827 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.177833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.177890 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.177982 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.178595 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.179342 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.180484 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.180853 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.180984 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.183840 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.183845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.183848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.183873 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.186037 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.186644 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.186652 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.186655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.186764 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.187102 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.187107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.187110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.187412 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.187552 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.187878 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.188196 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":60}
I20260927 22:53:45.193018 15683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Bootstrap starting.
I20260927 22:53:45.196060 15683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:45.197060 15683 log.cc:824] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:45.199662 15683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: No bootstrap required, opened a new log
I20260927 22:53:45.205353 15478 heartbeater.cc:503] Master 127.8.204.190:44551 was elected leader, sending a full tablet report...
I20260927 22:53:45.206799 15683 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } }
I20260927 22:53:45.207145 15683 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:45.207259 15683 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0c24fd1fec242dfaab8755016f0d8da, State: Initialized, Role: FOLLOWER
I20260927 22:53:45.207867 15683 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } }
I20260927 22:53:45.208635 15689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } } }
I20260927 22:53:45.208956 15689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:53:45.209594 15683 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:53:45.218129 15700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:53:45.218288 15700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:53:45.220575 15616 master_runner.cc:428] Detected that this master d0c24fd1fec242dfaab8755016f0d8da is joining an existing cluster
I20260927 22:53:45.220677 15616 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.204.189:45951
I20260927 22:53:45.241343 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.225012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.8.204.189:45951)
0927 22:53:45.225172 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.225178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.226163 (+ 985us) server_negotiation.cc:407] Connection header received
0927 22:53:45.226330 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.226334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.226389 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.226495 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.228287 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.229401 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.231724 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.232569 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.232679 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.235363 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.235372 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.235379 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.235423 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.238241 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.238863 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.238874 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.238878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.239009 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.239466 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.239472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.239477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.239903 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.240046 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.240565 (+ 519us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.240902 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":62}
I20260927 22:53:45.241350 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.224840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:44436)
0927 22:53:45.226065 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.226080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.226215 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.226701 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.226707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.227545 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.228134 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.228149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.229621 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.229628 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.231583 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.231600 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.233039 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.233047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.233257 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.233788 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.233817 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.235152 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.238367 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.238375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.238379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.238751 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.239134 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.239141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.239144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.239348 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.240138 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.240143 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.240300 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.240462 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.240778 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":132,"thread_start_us":922,"threads_started":1}
I20260927 22:53:45.242133 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.224709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.8.204.190:44551)
0927 22:53:45.224884 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.224891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.225673 (+ 782us) server_negotiation.cc:407] Connection header received
0927 22:53:45.225929 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.225936 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.226045 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.226119 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.228286 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.229420 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.231660 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.232301 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.232399 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.235124 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.235131 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.235136 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.235172 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.238524 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.239325 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.239331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.239335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.239475 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.239957 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.239964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.239969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.240318 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.240455 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.241535 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.241753 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:53:45.242686 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.224802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41710)
0927 22:53:45.225573 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.225595 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.225777 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.226570 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.226574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.227543 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.228120 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.228134 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.229640 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.229644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.231495 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.231513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.232664 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.232670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.232900 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.233600 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.233623 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.234910 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.238673 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.238678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.238682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.239150 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.239609 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.239614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.239618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.239806 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.240516 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.240525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.240722 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.240877 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.241190 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":57,"spinlock_wait_cycles":39680,"thread_start_us":502,"threads_started":1}
WARNING: no policy specified for oryx/127.8.204.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.188@KRBTEST.COM" created.
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549623, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.190@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
I20260927 22:53:45.266654 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.248957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:44438)
0927 22:53:45.249643 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.249682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.249893 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.250424 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.250432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.251265 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.251562 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.251577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.253209 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.253214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.254828 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.254843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.256055 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.256061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.256263 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.256879 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.256909 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.259812 (+ 2903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.263789 (+ 3977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.263805 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.263817 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.264331 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.264724 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.264729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.264733 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.264867 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.265431 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.265440 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.265644 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.265850 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.266193 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
I20260927 22:53:45.266712 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.248974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.8.204.189:45951)
0927 22:53:45.249160 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.249166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.249768 (+ 602us) server_negotiation.cc:407] Connection header received
0927 22:53:45.250122 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.250130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.250249 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.250352 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.251778 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.253042 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.254997 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.255864 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.256007 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.260017 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.260024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.260028 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.260063 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.263643 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.264456 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.264462 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.264466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.264609 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.264989 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.264996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.265002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.265298 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.265448 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.265983 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.266235 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:53:45.269443 15155 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.204.189:45951: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da"
member_type: NON_VOTER
last_known_addr {
host: "127.8.204.189"
port: 45951
}
attrs {
promote: true
}
}
dest_uuid: "36a20dcf2145413292db3c418e7e97ed"
cas_config_opid_index: -1
I20260927 22:53:45.270668 15155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } }
I20260927 22:53:45.280519 15713 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d0c24fd1fec242dfaab8755016f0d8da (127.8.204.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } }
I20260927 22:53:45.281670 15658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:45.285000 15713 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.204.189:45951
I20260927 22:53:45.285503 15658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Refusing update from remote peer 36a20dcf2145413292db3c418e7e97ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:53:45.285753 15616 master.cc:537] Master@127.8.204.189:45951 shutting down...
I20260927 22:53:45.286631 15713 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:45.287094 15712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } }
I20260927 22:53:45.287472 15712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
W20260927 22:53:45.291635 15139 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 22:53:45.296271 15139 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed -> Peer d0c24fd1fec242dfaab8755016f0d8da (127.8.204.189:45951): Couldn't send request to peer d0c24fd1fec242dfaab8755016f0d8da. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.188.keytab.
Entry for principal oryx/127.8.204.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.188.keytab.
I20260927 22:53:45.299335 15616 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:45.299535 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:45411
--webserver_interface=127.8.204.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.188.keytab
--principal=oryx/127.8.204.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:53:45.299765 15616 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:45.299935 15616 tablet_replica.cc:330] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: stopping tablet replica
I20260927 22:53:45.306044 15616 master.cc:552] Master@127.8.204.189:45951 shutdown complete.
I20260927 22:53:45.306135 15616 master_runner.cc:305] Clearing existing system tablet
I20260927 22:53:45.306607 15616 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:45.306680 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:45.309100 15616 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:53:45.311961 15718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:45.312443 15616 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:45.312685 15616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 22:53:45.312763 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
uuid: "d0c24fd1fec242dfaab8755016f0d8da"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "427e0b54a69e462c1ecd69f3f06d6f4f"
server_key_iv: "e43c8a42254ea57c85d46ebaf5ae1150"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.316275 15616 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:45.321831 15616 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:53:45.322038 15616 log.cc:1196] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:53:45.322467 15616 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Deleting consensus metadata
I20260927 22:53:45.322743 15616 master_runner.cc:315] Copying system tablet from 127.8.204.190:44551
I20260927 22:53:45.325172 15616 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: tablet copy: Beginning tablet copy session from remote peer at address 127.8.204.190:44551
I20260927 22:53:45.338737 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.325902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41724)
0927 22:53:45.327709 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.327739 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.327897 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.328312 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.328317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.328759 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.328953 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.328962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.330157 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.330162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.331088 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.331098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.331845 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.331850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.332012 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.332676 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.332693 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.333373 (+ 680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.336336 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.336343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.336347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.336679 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.337049 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.337053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.337056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.337179 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.337677 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.337682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.337866 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.338025 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.338296 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"thread_start_us":1600,"threads_started":1}
I20260927 22:53:45.338749 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.325925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.8.204.190:44551)
0927 22:53:45.326150 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.326160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.327833 (+ 1673us) server_negotiation.cc:407] Connection header received
0927 22:53:45.328051 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.328059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.328159 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.328302 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.329114 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.330022 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.331233 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.331685 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.331835 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.333568 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.333574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.333579 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.333619 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.336207 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.336802 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.336809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.336813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.336930 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.337280 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.337287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.337290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.337564 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.337728 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.338119 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.338310 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:53:45.339915 15175 tablet_copy_service.cc:140] P 36a20dcf2145413292db3c418e7e97ed: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d0c24fd1fec242dfaab8755016f0d8da ({username='oryx', principal='oryx/127.8.204.189@KRBTEST.COM'} at 127.0.0.1:41724)
I20260927 22:53:45.340199 15175 tablet_copy_service.cc:161] P 36a20dcf2145413292db3c418e7e97ed: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d0c24fd1fec242dfaab8755016f0d8da at {username='oryx', principal='oryx/127.8.204.189@KRBTEST.COM'} at 127.0.0.1:41724: session id = d0c24fd1fec242dfaab8755016f0d8da-00000000000000000000000000000000
I20260927 22:53:45.343086 15175 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:53:45.344964 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:53:45.349432 15616 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: tablet copy: Starting download of 0 data blocks...
I20260927 22:53:45.349763 15616 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: tablet copy: Starting download of 1 WAL segments...
I20260927 22:53:45.352041 15616 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:53:45.355011 15175 tablet_copy_service.cc:342] P 36a20dcf2145413292db3c418e7e97ed: Request end of tablet copy session d0c24fd1fec242dfaab8755016f0d8da-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.204.189@KRBTEST.COM'} at 127.0.0.1:41724
I20260927 22:53:45.355198 15175 tablet_copy_service.cc:433] P 36a20dcf2145413292db3c418e7e97ed: ending tablet copy session d0c24fd1fec242dfaab8755016f0d8da-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d0c24fd1fec242dfaab8755016f0d8da
I20260927 22:53:45.364975 15616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:45.367534 15725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:45.373667 15726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:45.380965 15728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:45.381421 15616 server_base.cc:1034] running on GCE node
I20260927 22:53:45.381909 15616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:45.382622 15616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:45.383824 15616 hybrid_clock.cc:648] HybridClock initialized: now 1790549625383746 us; error 50 us; skew 500 ppm
I20260927 22:53:45.384056 15616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:45.385921 15616 init.cc:377] Logged in from keytab as oryx/127.8.204.189@KRBTEST.COM (short username oryx)
I20260927 22:53:45.387123 15616 webserver.cc:460] Webserver started at http://127.8.204.189:35703/ using document root <none> and password file <none>
I20260927 22:53:45.387507 15616 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:45.387598 15616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:45.389806 15616 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 22:53:45.391563 15735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:45.391939 15616 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 22:53:45.392086 15616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
uuid: "d0c24fd1fec242dfaab8755016f0d8da"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "427e0b54a69e462c1ecd69f3f06d6f4f"
server_key_iv: "e43c8a42254ea57c85d46ebaf5ae1150"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.392242 15616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:45.401551 15616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:45.404971 15616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:45.433990 15616 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.189:45951
I20260927 22:53:45.434031 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.189:45951 every 8 connection(s)
I20260927 22:53:45.434916 15616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
I20260927 22:53:45.441740 15787 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:53:45.444160 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Bootstrap starting.
I20260927 22:53:45.455927 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:53:45.456387 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Bootstrap complete.
I20260927 22:53:45.457176 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } }
I20260927 22:53:45.457391 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0c24fd1fec242dfaab8755016f0d8da, State: Initialized, Role: LEARNER
I20260927 22:53:45.457615 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } }
I20260927 22:53:45.458348 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } }
I20260927 22:53:45.458649 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:45.458748 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: LEARNER
I20260927 22:53:45.466132 15801 catalog_manager.cc:1259] Loaded cluster ID: d55fb42e8da6423e8e026d2983783b7a
I20260927 22:53:45.466241 15801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: loading cluster ID for follower catalog manager: success
I20260927 22:53:45.468556 15801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: acquiring CA information for follower catalog manager: success
I20260927 22:53:45.470216 15801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:53:45.679819 15611 heartbeater.cc:503] Master 127.8.204.190:44551 was elected leader, sending a full tablet report...
I20260927 22:53:45.684676 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.675417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:44444)
0927 22:53:45.675682 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.675701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.675819 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.677203 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.677208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.677242 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:45.677421 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.677431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.679447 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.679452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.682000 (+ 2548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.682016 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.683818 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.683825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.684046 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.684111 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.684213 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.684367 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 22:53:45.684996 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.676159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.8.204.189:45951)
0927 22:53:45.676823 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.676829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.676851 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:53:45.676920 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.676925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.677017 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.677146 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:45.677580 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.679261 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.682177 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.683652 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.683773 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.684035 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.684249 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.684382 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":319,"threads_started":1}
I20260927 22:53:45.686234 15762 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 22:53:45.687065 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a20dcf2145413292db3c418e7e97ed. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } }
I20260927 22:53:45.687364 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: LEARNER
W20260927 22:53:45.697253 15716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:45.697566 15716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:45.697737 15716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:45.705854 15716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:45.705994 15716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:45.706063 15716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:45.706156 15716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:45.706197 15716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:45.716886 15716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.188.keytab
--principal=oryx/127.8.204.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:45.718181 15716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:45.719661 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:45.737668 15807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:45.737591 15808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:45.737651 15810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:45.739552 15716 server_base.cc:1034] running on GCE node
I20260927 22:53:45.740398 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:45.742005 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:45.743234 15716 hybrid_clock.cc:648] HybridClock initialized: now 1790549625743179 us; error 47 us; skew 500 ppm
I20260927 22:53:45.743594 15716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:45.746896 15716 init.cc:377] Logged in from keytab as oryx/127.8.204.188@KRBTEST.COM (short username oryx)
I20260927 22:53:45.748633 15716 webserver.cc:460] Webserver started at http://127.8.204.188:42523/ using document root <none> and password file <none>
I20260927 22:53:45.749369 15716 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:45.749516 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:45.749807 15716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:45.752719 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/instance:
uuid: "79389ed8503b43068a63586589802748"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "0f6260bd8a4a8fa86917ad5a8fa6feaf"
server_key_iv: "4e29cbacb3ff1ec495722102cbbab4d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.753419 15716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal/instance:
uuid: "79389ed8503b43068a63586589802748"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "0f6260bd8a4a8fa86917ad5a8fa6feaf"
server_key_iv: "4e29cbacb3ff1ec495722102cbbab4d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.758500 15716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:53:45.761996 15817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:45.763355 15716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 22:53:45.763548 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
uuid: "79389ed8503b43068a63586589802748"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "0f6260bd8a4a8fa86917ad5a8fa6feaf"
server_key_iv: "4e29cbacb3ff1ec495722102cbbab4d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:45.763737 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:45.782620 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:45.787792 15716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:45.788014 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:45.806948 15716 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.188:45411
I20260927 22:53:45.806954 15868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.188:45411 every 8 connection(s)
I20260927 22:53:45.809075 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
I20260927 22:53:45.813675 15869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:45.818643 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15716
I20260927 22:53:45.821966 15869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.188@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.188@KRBTEST.COM
I20260927 22:53:45.840457 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.825866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.8.204.190:44551)
0927 22:53:45.826068 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.826073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.826446 (+ 373us) server_negotiation.cc:407] Connection header received
0927 22:53:45.826633 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.826638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.826713 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.826821 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.828083 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.828933 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.830552 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.830952 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.831040 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.834317 (+ 3277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.834324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.834329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.834367 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.837246 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.838108 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.838112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.838114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.838229 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.838679 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.838683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.838686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.839036 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.839186 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.839981 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.840136 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:53:45.841511 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.825612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41736)
0927 22:53:45.826275 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.826328 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.826499 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.826894 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.826899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.827603 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.827898 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.827911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.829066 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.829072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.830400 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.830417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.831164 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.831173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.831398 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.831969 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.831994 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.834119 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.837371 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.837379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.837392 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.837991 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.838339 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.838347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.838351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.838563 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.839159 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.839164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.839606 (+ 442us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.839898 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.840402 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260927 22:53:45.841853 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.820918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.8.204.188:45411)
0927 22:53:45.822235 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.822247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.822289 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:53:45.823988 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.824011 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.824469 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.824845 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.826638 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.828149 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.829668 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.830333 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.830454 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.834900 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.834907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.834917 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.834963 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.838542 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.839321 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.839328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.839331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.839472 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.839804 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.839809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.839811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.840185 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.840442 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.841267 (+ 825us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.841426 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":402,"threads_started":1}
I20260927 22:53:45.843411 15869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.188@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
I20260927 22:53:45.855414 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.844164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:44454)
0927 22:53:45.844558 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.844574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.844671 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.844921 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.844924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.845202 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.845320 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.845326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.846250 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.846253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.847140 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.847150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.847842 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.847845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.847979 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.848463 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.848482 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.850400 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.853561 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.853565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.853567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.853900 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.854204 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.854207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.854209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.854320 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.854772 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.854775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.854874 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.854971 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.855145 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260927 22:53:45.855490 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.844221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.8.204.189:45951)
0927 22:53:45.844408 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.844412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.844659 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:53:45.844750 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.844754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.844814 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.844891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.845431 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.846137 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.847324 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.847726 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.847818 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.850567 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.850571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.850575 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.850602 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.853458 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.853998 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.854002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.854004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.854116 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.854401 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.854404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.854406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.854680 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.854785 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.855046 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.855196 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:53:45.856537 15869 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549625, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.188@KRBTEST.COM for oryx/127.8.204.188@KRBTEST.COM
I20260927 22:53:45.868426 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.857264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:42670)
0927 22:53:45.857526 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.857544 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.857674 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.857949 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.857953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.858319 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.858443 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.858450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.859430 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.859434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.860377 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.860388 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.861093 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.861097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.861299 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.861936 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.861954 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.863795 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.866255 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.866258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.866261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.866687 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.867019 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.867022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.867024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.867155 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.867689 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.867693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.867807 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.867916 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.868149 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:53:45.868507 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.857239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.8.204.188:45411)
0927 22:53:45.857435 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.857446 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.857611 (+ 165us) server_negotiation.cc:407] Connection header received
0927 22:53:45.857780 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.857785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.857844 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.857949 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.858603 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.859238 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.860520 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.860973 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.861073 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.863953 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.863957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.863961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.863998 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.866140 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.866801 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.866805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.866807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.866914 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.867240 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.867243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.867245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.867592 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.867692 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.868006 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.868264 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":77}
I20260927 22:53:45.872805 15869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Bootstrap starting.
I20260927 22:53:45.875773 15869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:45.877077 15869 log.cc:824] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:45.879570 15869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: No bootstrap required, opened a new log
I20260927 22:53:45.886137 15869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } }
I20260927 22:53:45.886399 15869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:45.886503 15869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79389ed8503b43068a63586589802748, State: Initialized, Role: FOLLOWER
I20260927 22:53:45.886958 15869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } }
I20260927 22:53:45.887699 15875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } } }
I20260927 22:53:45.888036 15875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:53:45.888651 15869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:45.896385 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15216
W20260927 22:53:45.898066 15886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:53:45.898186 15886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:53:45.902436 15716 master_runner.cc:428] Detected that this master 79389ed8503b43068a63586589802748 is joining an existing cluster
I20260927 22:53:45.902556 15716 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.204.188:45411
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.129@KRBTEST.COM
I20260927 22:53:45.922053 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.897981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.8.204.129:41189)
0927 22:53:45.898737 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.898743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.898764 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:45.898815 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.898820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.898985 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.899230 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.900503 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.901964 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.903867 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.904717 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.904837 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.911929 (+ 7092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.911944 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.911960 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.912003 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.916307 (+ 4304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.917767 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.917780 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.917786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.917991 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.918500 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.918507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.918513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.919054 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.920304 (+ 1250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.921121 (+ 817us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.921450 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":468,"threads_started":1}
I20260927 22:53:45.923527 15339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:45.934612 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.911462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42674 (local address 127.8.204.188:45411)
0927 22:53:45.911645 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.911650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.915863 (+ 4213us) server_negotiation.cc:407] Connection header received
0927 22:53:45.916144 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.916151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.916227 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.916292 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.919774 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.920879 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.923051 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.923824 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.923917 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.928053 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.928059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.928063 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.928108 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.931738 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.932477 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.932482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.932485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.932618 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.933011 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.933016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.933019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.933378 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.933521 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.934022 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.934238 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:53:45.935608 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.911030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:42674)
0927 22:53:45.915763 (+ 4733us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.915779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.916009 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.916470 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.916477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.919233 (+ 2756us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.919631 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.919645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.921149 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.921154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.922910 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.922923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.924202 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.924208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.924406 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.925051 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.925107 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.927715 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.931884 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.931888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.931894 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.932360 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.932755 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.932762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.932766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.932908 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.933586 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.933590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.933739 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.934829 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.935076 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4597,"thread_start_us":5489,"threads_started":1}
I20260927 22:53:45.947690 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.911079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.8.204.189:45951)
0927 22:53:45.911260 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.911288 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.923065 (+ 11777us) server_negotiation.cc:407] Connection header received
0927 22:53:45.923879 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.923888 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.923985 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.924068 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.929019 (+ 4951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.930478 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.937290 (+ 6812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.938180 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.938304 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.941270 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.941276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.941281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.941318 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.944625 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.945460 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.945465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.945469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.945611 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.946031 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.946036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.946039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.946356 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.946503 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.947047 (+ 544us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.947361 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:53:45.948664 15875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:45.949242 15875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } }
I20260927 22:53:45.951254 15875 leader_election.cc:288] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551), d0c24fd1fec242dfaab8755016f0d8da (127.8.204.189:45951)
I20260927 22:53:45.951948 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.911299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:44458)
0927 22:53:45.922965 (+ 11666us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.922984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.923691 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.924322 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.924330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.925155 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.925634 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.925649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.930736 (+ 5087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.930743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.932943 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.932959 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.939207 (+ 6248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.939214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.939405 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.940005 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.940026 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.941068 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.944800 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.944805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.944808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.945313 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.945749 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.945754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.945757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.945910 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.946585 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.946590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.946781 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.947889 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.948191 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11541,"thread_start_us":4497,"threads_started":1}
I20260927 22:53:45.951751 15165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79389ed8503b43068a63586589802748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36a20dcf2145413292db3c418e7e97ed" is_pre_election: true
I20260927 22:53:45.951834 15762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79389ed8503b43068a63586589802748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0c24fd1fec242dfaab8755016f0d8da" is_pre_election: true
I20260927 22:53:45.953294 15821 leader_election.cc:302] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79389ed8503b43068a63586589802748; no voters: 36a20dcf2145413292db3c418e7e97ed, d0c24fd1fec242dfaab8755016f0d8da
I20260927 22:53:45.954025 15875 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:53:45.954216 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.910713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41742 (local address 127.8.204.190:44551)
0927 22:53:45.910914 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.910921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.915234 (+ 4313us) server_negotiation.cc:407] Connection header received
0927 22:53:45.915480 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.915486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.915579 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.915683 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.925857 (+ 10174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.927313 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.934312 (+ 6999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.935185 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.935329 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.940271 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.940281 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.940288 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.940341 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.945185 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.946926 (+ 1741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.946934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.946940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.947164 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.947862 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.947869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.947875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.948270 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.948422 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.953650 (+ 5228us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.953874 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:53:45.955152 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.910942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41742)
0927 22:53:45.915121 (+ 4179us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.915141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.915305 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.916630 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.916636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.918620 (+ 1984us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.923487 (+ 4867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.923505 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.927582 (+ 4077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.927590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.934129 (+ 6539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.934145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.935634 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.935642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.935903 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.936678 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.936719 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.940014 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.945366 (+ 5352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.945374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.945379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.945977 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.947404 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.947412 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.947418 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.947649 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.948599 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.948606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.953253 (+ 4647us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.954424 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.954752 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4040,"thread_start_us":11172,"threads_started":1}
Sep 27 22:53:45 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549623, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.190@KRBTEST.COM for oryx/127.8.204.188@KRBTEST.COM
I20260927 22:53:45.976779 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.960551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:42684)
0927 22:53:45.960836 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:45.960857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:45.960993 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:45.961471 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:45.961477 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:45.961919 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:45.962157 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.962169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.963890 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.963894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.965273 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:45.965287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.966535 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.966541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:45.966704 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.967412 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:45.967435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:45.970140 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:45.974075 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.974080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.974094 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.974543 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.974946 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:45.974950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:45.974953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:45.975108 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:45.975801 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:45.975808 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:45.975968 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 22:53:45.976124 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 22:53:45.976395 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260927 22:53:45.977643 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:45.960526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42684 (local address 127.8.204.188:45411)
0927 22:53:45.960688 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:45.960694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:45.961109 (+ 415us) server_negotiation.cc:407] Connection header received
0927 22:53:45.961210 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:45.961217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:45.961311 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:45.961422 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:45.962314 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.963711 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.965680 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:45.966353 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:45.966494 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:45.970344 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:45.970353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:45.970358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:45.970392 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:45.973925 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.974681 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.974686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.974690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.974818 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:45.975233 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:45.975238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:45.975241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:45.975612 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:45.975744 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:45.977057 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0927 22:53:45.977281 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:53:45.978962 15155 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.204.188:45411: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "79389ed8503b43068a63586589802748"
member_type: NON_VOTER
last_known_addr {
host: "127.8.204.188"
port: 45411
}
attrs {
promote: true
}
}
dest_uuid: "36a20dcf2145413292db3c418e7e97ed"
cas_config_opid_index: 5
I20260927 22:53:45.980007 15155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } }
I20260927 22:53:45.981979 15712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 79389ed8503b43068a63586589802748 (127.8.204.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:45.985968 15712 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.204.188:45411
I20260927 22:53:45.986491 15716 master.cc:537] Master@127.8.204.188:45411 shutting down...
I20260927 22:53:45.987941 15762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 LEARNER]: Refusing update from remote peer 36a20dcf2145413292db3c418e7e97ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260927 22:53:46.011976 15141 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed -> Peer 79389ed8503b43068a63586589802748 (127.8.204.188:45411): Couldn't send request to peer 79389ed8503b43068a63586589802748. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 22:53:46.013480 15897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:53:46.017697 15716 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:46.018175 15716 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:46.018420 15716 tablet_replica.cc:330] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: stopping tablet replica
I20260927 22:53:46.026356 15716 master.cc:552] Master@127.8.204.188:45411 shutdown complete.
I20260927 22:53:46.026469 15716 master_runner.cc:305] Clearing existing system tablet
I20260927 22:53:46.026847 15716 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:46.026944 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:46.026811 15762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 79389ed8503b43068a63586589802748 (127.8.204.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.032248 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.032521 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: LEARNER
I20260927 22:53:46.034309 15716 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:53:46.034109 15898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.034477 15898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:46.040241 15900 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: attempting to promote NON_VOTER d0c24fd1fec242dfaab8755016f0d8da to VOTER
I20260927 22:53:46.041059 15900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } }
I20260927 22:53:46.044986 15762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 LEARNER]: Refusing update from remote peer 36a20dcf2145413292db3c418e7e97ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:53:46.045569 15897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:53:46.047089 15904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:46.047719 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.047183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:42698)
0927 22:53:46.047431 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.047547 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 22:53:46.048776 15762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0c24fd1fec242dfaab8755016f0d8da (127.8.204.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.053653 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.053199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:42708)
0927 22:53:46.053393 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.053471 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260927 22:53:46.053825 15141 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed -> Peer 79389ed8503b43068a63586589802748 (127.8.204.188:45411): Couldn't send request to peer 79389ed8503b43068a63586589802748. Status: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:53:46.054319 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.054553 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:53:46.054867 15897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: NON_VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: true } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.055224 15897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:46.055671 15716 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:46.056000 15716 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.004s sys 0.008s
I20260927 22:53:46.055882 15712 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d0c24fd1fec242dfaab8755016f0d8da (127.8.204.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.056121 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
uuid: "79389ed8503b43068a63586589802748"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "0f6260bd8a4a8fa86917ad5a8fa6feaf"
server_key_iv: "4e29cbacb3ff1ec495722102cbbab4d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:46.060053 15900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.060149 15897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.060436 15897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:46.060598 15900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:46.062045 15716 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:46.068987 15716 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:53:46.069134 15716 log.cc:1196] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:53:46.069988 15716 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Deleting consensus metadata
I20260927 22:53:46.070215 15716 master_runner.cc:315] Copying system tablet from 127.8.204.190:44551
I20260927 22:53:46.095677 15716 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: tablet copy: Beginning tablet copy session from remote peer at address 127.8.204.190:44551
I20260927 22:53:46.101624 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15216
I20260927 22:53:46.113327 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.096540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.8.204.190:44551)
0927 22:53:46.096753 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:46.096760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:46.101088 (+ 4328us) server_negotiation.cc:407] Connection header received
0927 22:53:46.101420 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:46.101425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:46.101510 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:46.101640 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:46.102767 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.104101 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.105502 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.105893 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.106036 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.107921 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:46.107928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:46.107932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:46.107979 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:46.110541 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.111243 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.111249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.111252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.111438 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.111850 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.111855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.111859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.112222 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:46.112364 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:46.112742 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 22:53:46.112976 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:53:46.113356 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.096506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41750)
0927 22:53:46.100953 (+ 4447us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.100973 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:46.101266 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:46.101666 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:46.101671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:46.102231 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:46.102589 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.102605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.104286 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.104290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.105319 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.105329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.106051 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.106057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.106232 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.106906 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:46.106923 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:46.107733 (+ 810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:46.110672 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.110679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.110683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.111126 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.111543 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.111552 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.111555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.111707 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.112329 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:46.112334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:46.112505 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 22:53:46.112648 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:53:46.112945 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4291,"thread_start_us":3537,"threads_started":1}
I20260927 22:53:46.113845 15175 tablet_copy_service.cc:140] P 36a20dcf2145413292db3c418e7e97ed: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 79389ed8503b43068a63586589802748 ({username='oryx', principal='oryx/127.8.204.188@KRBTEST.COM'} at 127.0.0.1:41750)
I20260927 22:53:46.113986 15175 tablet_copy_service.cc:161] P 36a20dcf2145413292db3c418e7e97ed: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 79389ed8503b43068a63586589802748 at {username='oryx', principal='oryx/127.8.204.188@KRBTEST.COM'} at 127.0.0.1:41750: session id = 79389ed8503b43068a63586589802748-00000000000000000000000000000000
I20260927 22:53:46.116510 15175 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:53:46.118022 15716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:53:46.122956 15716 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: tablet copy: Starting download of 0 data blocks...
I20260927 22:53:46.123224 15716 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: tablet copy: Starting download of 1 WAL segments...
I20260927 22:53:46.124605 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:41189
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:44551
--webserver_port=40123
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:53:46.125633 15716 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:53:46.130189 15175 tablet_copy_service.cc:342] P 36a20dcf2145413292db3c418e7e97ed: Request end of tablet copy session 79389ed8503b43068a63586589802748-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.204.188@KRBTEST.COM'} at 127.0.0.1:41750
I20260927 22:53:46.130309 15175 tablet_copy_service.cc:433] P 36a20dcf2145413292db3c418e7e97ed: ending tablet copy session 79389ed8503b43068a63586589802748-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 79389ed8503b43068a63586589802748
I20260927 22:53:46.142410 15716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:46.145267 15913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:46.145689 15914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:46.154155 15716 server_base.cc:1034] running on GCE node
W20260927 22:53:46.155222 15917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:46.155778 15716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:46.156383 15716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:46.157670 15716 hybrid_clock.cc:648] HybridClock initialized: now 1790549626157612 us; error 33 us; skew 500 ppm
I20260927 22:53:46.157887 15716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:46 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549626, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:46.160126 15716 init.cc:377] Logged in from keytab as oryx/127.8.204.188@KRBTEST.COM (short username oryx)
I20260927 22:53:46.164180 15716 webserver.cc:460] Webserver started at http://127.8.204.188:36261/ using document root <none> and password file <none>
I20260927 22:53:46.164562 15716 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:46.164711 15716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:46.167757 15716 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:53:46.169622 15923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:46.170025 15716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 22:53:46.170176 15716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
uuid: "79389ed8503b43068a63586589802748"
format_stamp: "Formatted at 2026-09-27 22:53:45 on dist-test-slave-40vx"
server_key: "0f6260bd8a4a8fa86917ad5a8fa6feaf"
server_key_iv: "4e29cbacb3ff1ec495722102cbbab4d9"
server_key_version: "encryptionkey@0"
I20260927 22:53:46.170399 15716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:46.205449 15716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:46.210008 15716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:46.240203 15716 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.188:45411
I20260927 22:53:46.240258 15974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.188:45411 every 8 connection(s)
I20260927 22:53:46.241416 15716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
I20260927 22:53:46.246793 15975 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:53:46.250474 15975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Bootstrap starting.
I20260927 22:53:46.269550 15975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:53:46.270097 15975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: Bootstrap complete.
I20260927 22:53:46.271150 15975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } }
I20260927 22:53:46.271452 15975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 79389ed8503b43068a63586589802748, State: Initialized, Role: LEARNER
I20260927 22:53:46.271711 15975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } }
I20260927 22:53:46.272804 15975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:46.272550 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.273069 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:53:46.284207 15989 catalog_manager.cc:1259] Loaded cluster ID: d55fb42e8da6423e8e026d2983783b7a
I20260927 22:53:46.284406 15989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: loading cluster ID for follower catalog manager: success
I20260927 22:53:46.287904 15989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: acquiring CA information for follower catalog manager: success
I20260927 22:53:46.290287 15989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:53:46.503504 15912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:46.503811 15912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:46.503983 15912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:46.510902 15912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:46.511042 15912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:46.511225 15912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:46.520974 15912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:41189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=40123
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:46.522687 15912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:46.524019 15912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:46.537290 15995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:46.537642 15994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:46.537365 15997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:46.538257 15912 server_base.cc:1034] running on GCE node
I20260927 22:53:46.538841 15912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:46.539780 15912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:46.541013 15912 hybrid_clock.cc:648] HybridClock initialized: now 1790549626540943 us; error 51 us; skew 500 ppm
I20260927 22:53:46.541409 15912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:46 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549626, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:46.545109 15912 init.cc:377] Logged in from keytab as oryx/127.8.204.129@KRBTEST.COM (short username oryx)
I20260927 22:53:46.546496 15912 webserver.cc:460] Webserver started at http://127.8.204.129:40123/ using document root <none> and password file <none>
I20260927 22:53:46.547184 15912 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:46.547358 15912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:46.552597 15912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:53:46.555258 16004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:46.556692 15912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:46.556874 15912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "58fd2a85f6534816b8a77be4c332c34a"
format_stamp: "Formatted at 2026-09-27 22:53:43 on dist-test-slave-40vx"
server_key: "8bc3a0b2f21ad52c40cad34cf130c500"
server_key_iv: "17eecb7c840e0393ba2e3613bf3f4a79"
server_key_version: "encryptionkey@0"
I20260927 22:53:46.557879 15912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:46.584108 15912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:46.587926 15912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:46.588194 15912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:46.589223 15912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:46.590539 15912 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:46.590654 15912 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:46.590781 15912 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:46.590865 15912 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:46.624343 15912 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41189
I20260927 22:53:46.624382 16116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41189 every 8 connection(s)
I20260927 22:53:46.625803 15912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:46.632071 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 15912
I20260927 22:53:46.632459 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15350
Sep 27 22:53:46 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549626, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.188@KRBTEST.COM
Sep 27 22:53:46 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549626, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
Sep 27 22:53:46 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.130@KRBTEST.COM
Sep 27 22:53:46 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549626, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
I20260927 22:53:46.677856 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.628655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:35043 (local address 127.8.204.188:45411)
0927 22:53:46.653052 (+ 24397us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:46.653061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:46.653087 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:53:46.653163 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:46.653171 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:46.653274 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:46.655575 (+ 2301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:46.655834 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.657715 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.661023 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.661949 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.662098 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.667717 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:46.667726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:46.667731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:46.667770 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:46.671170 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.672619 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.672625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.672629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.672759 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.673853 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.673858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.673862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.674229 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:46.674366 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:46.677343 (+ 2977us) server_negotiation.cc:299] Negotiation successful
0927 22:53:46.677549 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24228,"thread_start_us":24717,"threads_started":1}
I20260927 22:53:46.678380 16124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.630349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:37743)
0927 22:53:46.643365 (+ 13016us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.643386 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:46.648523 (+ 5137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:46.655717 (+ 7194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:46.655722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:46.656646 (+ 924us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:46.656896 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.656907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.660371 (+ 3464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.660376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.662002 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.662015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.664097 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.664105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.664282 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.664872 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:46.664898 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:46.667865 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:46.671845 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.671852 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.671858 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.672350 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.674769 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.674775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.674780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.674982 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.675641 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:46.675646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:46.675990 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 22:53:46.676289 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 22:53:46.676913 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12850,"thread_start_us":9037,"threads_started":1}
I20260927 22:53:46.678399 16123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.629409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.8.204.129:35043)
0927 22:53:46.637356 (+ 7947us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.637412 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:46.649471 (+ 12059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:46.653584 (+ 4113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:46.653592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:46.654306 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:46.655336 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.655369 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.657929 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.657938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.660754 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.660774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.662578 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.662588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.662962 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.663715 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:46.663749 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:46.666953 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:46.671346 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.671355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.671373 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.672500 (+ 1127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.673448 (+ 948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.673453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.673456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.673712 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.674436 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:46.674444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:46.676908 (+ 2464us) client_negotiation.cc:769] Sending connection context
0927 22:53:46.677051 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 22:53:46.677533 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7612,"thread_start_us":7532,"threads_started":1}
I20260927 22:53:46.679631 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.631233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:37743 (local address 127.8.204.190:44551)
0927 22:53:46.640434 (+ 9201us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:46.640443 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:46.643557 (+ 3114us) server_negotiation.cc:407] Connection header received
0927 22:53:46.655182 (+ 11625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:46.655188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:46.655292 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:46.657058 (+ 1766us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:46.657214 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.658268 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.662248 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.662959 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.664002 (+ 1043us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.668038 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:46.668045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:46.668051 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:46.668102 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:46.671707 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.673156 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.673162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.673167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.673320 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.675123 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.675127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.675131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.675483 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:46.678823 (+ 3340us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:46.678945 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 22:53:46.679180 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9053,"thread_start_us":8939,"threads_started":1}
I20260927 22:53:46.679885 16128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.633570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59122 (local address 127.8.204.130:42231)
0927 22:53:46.647486 (+ 13916us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:46.647494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:46.647523 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:53:46.647598 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:46.647606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:46.648056 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:46.648225 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:46.656498 (+ 8273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.658044 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.660637 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.661482 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.661610 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.668526 (+ 6916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:46.668544 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:46.668563 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:46.668626 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:46.674510 (+ 5884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.676116 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.676124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.676128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.676311 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.676800 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.676805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.676808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.677337 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:46.677611 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:46.678547 (+ 936us) server_negotiation.cc:299] Negotiation successful
0927 22:53:46.679363 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13753,"thread_start_us":13587,"threads_started":1}
I20260927 22:53:46.680508 16117 heartbeater.cc:347] Connected to a master server at 127.8.204.188:45411
I20260927 22:53:46.680815 16117 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:46.681423 15473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:46.681680 16117 heartbeater.cc:511] Master 127.8.204.188:45411 requested a full tablet report, sending...
I20260927 22:53:46.681731 16118 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44551
I20260927 22:53:46.681838 16118 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:46.682282 16118 heartbeater.cc:511] Master 127.8.204.190:44551 requested a full tablet report, sending...
I20260927 22:53:46.683544 15154 ts_manager.cc:194] Re-registered known tserver with Master: 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189)
I20260927 22:53:46.684096 15940 ts_manager.cc:194] Registered new tserver with Master: 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189)
I20260927 22:53:46.684824 15154 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:37743
I20260927 22:53:46.685491 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.648111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:42722)
0927 22:53:46.663210 (+ 15099us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.663232 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:46.663539 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:46.672926 (+ 9387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:46.672930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:46.672956 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:46.673304 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.673316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.674700 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.674707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.676635 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.676649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.681849 (+ 5200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.681855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.684189 (+ 2334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.684284 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 22:53:46.684390 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:53:46.684595 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14956,"mutex_wait_us":8,"thread_start_us":15177,"threads_started":1}
I20260927 22:53:46.686473 16130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.648257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.8.204.188:45411)
0927 22:53:46.672631 (+ 24374us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:46.672636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:46.672651 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:53:46.672707 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:46.672711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:46.672797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:46.672910 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:46.673448 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.674599 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.680290 (+ 5691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.681681 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.681824 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.682039 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:46.686039 (+ 4000us) server_negotiation.cc:299] Negotiation successful
0927 22:53:46.686167 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24238,"thread_start_us":25478,"threads_started":1}
I20260927 22:53:46.688227 15950 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 22:53:46.689755 16122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.628722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:58125 (local address 127.8.204.189:45951)
0927 22:53:46.642780 (+ 14058us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:46.642789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:46.642821 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:53:46.645811 (+ 2990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:46.645819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:46.645937 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:46.646060 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:46.664257 (+ 18197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.665543 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.672286 (+ 6743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.672945 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.673076 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.678377 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:46.678386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:46.678391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:46.678433 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:46.686267 (+ 7834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.687363 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.687369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.687372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.687511 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:46.687934 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:46.687942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:46.687948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:46.688281 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:46.688430 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:46.689159 (+ 729us) server_negotiation.cc:299] Negotiation successful
0927 22:53:46.689362 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13900,"thread_start_us":13767,"threads_started":1}
I20260927 22:53:46.689780 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:46.629409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.8.204.129:58125)
0927 22:53:46.639641 (+ 10232us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:46.639663 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:46.645113 (+ 5450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:46.646636 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:46.646648 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:46.659868 (+ 13220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:46.660163 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.660175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.665742 (+ 5567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.665749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.672151 (+ 6402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:46.672165 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:46.673331 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:46.673340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:46.673515 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:46.674081 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:46.674114 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:46.677314 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:46.686486 (+ 9172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.686491 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.686497 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.687179 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.687667 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:46.687674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:46.687681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:46.687812 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:46.688410 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:46.688414 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:46.688908 (+ 494us) client_negotiation.cc:769] Sending connection context
0927 22:53:46.689046 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:53:46.689342 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10070,"mutex_wait_us":43,"spinlock_wait_cycles":66304,"thread_start_us":13376,"threads_started":1}
I20260927 22:53:46.689755 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a20dcf2145413292db3c418e7e97ed. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:46.690073 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:53:46.690723 16119 heartbeater.cc:347] Connected to a master server at 127.8.204.189:45951
I20260927 22:53:46.690862 16119 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:46.691236 16119 heartbeater.cc:511] Master 127.8.204.189:45951 requested a full tablet report, sending...
I20260927 22:53:46.692926 15752 ts_manager.cc:194] Registered new tserver with Master: 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189)
I20260927 22:53:46.774544 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15350
I20260927 22:53:46.791762 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:42231
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:44551
--webserver_port=32925
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:53:47.027024 15898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: NON_VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: true } } }
I20260927 22:53:47.027472 15898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:47.029187 15898 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: attempting to promote NON_VOTER 79389ed8503b43068a63586589802748 to VOTER
I20260927 22:53:47.029745 15898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } }
I20260927 22:53:47.031545 15950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 LEARNER]: Refusing update from remote peer 36a20dcf2145413292db3c418e7e97ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:53:47.031615 15762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Refusing update from remote peer 36a20dcf2145413292db3c418e7e97ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:53:47.032199 15898 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:53:47.032660 16134 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:53:47.034854 16134 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79389ed8503b43068a63586589802748 (127.8.204.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.035524 15762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79389ed8503b43068a63586589802748 (127.8.204.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.038084 16133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.038457 16133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:47.038753 16134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.038839 15898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.039017 16134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:47.039070 15898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:47.039891 15950 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 79389ed8503b43068a63586589802748 (127.8.204.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.042929 15978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.043174 15978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:53:47.043749 16137 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36a20dcf2145413292db3c418e7e97ed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:47.044173 16137 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: FOLLOWER
W20260927 22:53:47.124593 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:47.124843 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:47.125002 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:47.132411 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:47.132543 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:47.132723 16132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:47.143019 16132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:42231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=32925
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:47.144378 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:47.145732 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:47.158965 16142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:47.159006 16143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:47.159029 16145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:47.159883 16132 server_base.cc:1034] running on GCE node
I20260927 22:53:47.160465 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:47.161430 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:47.162657 16132 hybrid_clock.cc:648] HybridClock initialized: now 1790549627162607 us; error 57 us; skew 500 ppm
I20260927 22:53:47.163002 16132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:47.166033 16132 init.cc:377] Logged in from keytab as oryx/127.8.204.130@KRBTEST.COM (short username oryx)
I20260927 22:53:47.167395 16132 webserver.cc:460] Webserver started at http://127.8.204.130:32925/ using document root <none> and password file <none>
I20260927 22:53:47.168061 16132 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:47.168210 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:47.173555 16132 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:53:47.176198 16152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:47.177542 16132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:53:47.177706 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "0227056099ad4354bde4a3cae8940c96"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "1f100037c82a4d2b6467c8cec84d33c8"
server_key_iv: "77b5eb254ff5b566933469561b8ce6f4"
server_key_version: "encryptionkey@0"
I20260927 22:53:47.178654 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:47.192337 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:47.196396 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:47.196623 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:47.197475 16132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:47.198787 16132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:47.198869 16132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:47.198935 16132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:47.199000 16132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:47.231875 16132 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:42231
I20260927 22:53:47.231905 16264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:42231 every 8 connection(s)
I20260927 22:53:47.233253 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:47.242256 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 16132
I20260927 22:53:47.242520 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15483
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.131@KRBTEST.COM
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.188@KRBTEST.COM
I20260927 22:53:47.278757 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.236119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:53385 (local address 127.8.204.189:45951)
0927 22:53:47.248406 (+ 12287us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.248416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.259809 (+ 11393us) server_negotiation.cc:407] Connection header received
0927 22:53:47.259970 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.259974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.260046 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.260161 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.261359 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.262455 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.264989 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.265600 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.265715 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.271410 (+ 5695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.271423 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.271426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.271469 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.274972 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.276015 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.276024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.276029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.276163 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.276911 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.276915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.276917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.277302 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.277462 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.278188 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.278406 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12127,"thread_start_us":12293,"threads_started":1}
I20260927 22:53:47.280149 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.239209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.8.204.130:53385)
0927 22:53:47.259703 (+ 20494us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.259717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.259850 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.260219 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.260223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.260788 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:47.261133 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.261146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.262637 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.262641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.264796 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.264816 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.266397 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.266404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.266740 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.267699 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:47.267724 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:47.271216 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:47.275141 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.275153 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.275174 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.275862 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.276304 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.276309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.276313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.276592 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.277429 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:47.277437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:47.277818 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.278091 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.278615 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20364,"mutex_wait_us":65,"thread_start_us":17434,"threads_started":1}
I20260927 22:53:47.281468 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.237695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:33095)
0927 22:53:47.256920 (+ 19225us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.256946 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.257072 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.258361 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.258378 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.260200 (+ 1822us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:47.261095 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.261121 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.263377 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.263381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.264974 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.264983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.266597 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.266602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.266743 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.267694 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:47.267732 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:47.270816 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:47.275879 (+ 5063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.275883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.275887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.276439 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.279164 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.279170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.279174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.279356 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.280477 (+ 1121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:47.280481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:47.280640 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.280798 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.281063 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19084,"mutex_wait_us":54,"spinlock_wait_cycles":36864,"thread_start_us":14310,"threads_started":1}
I20260927 22:53:47.281903 16267 heartbeater.cc:347] Connected to a master server at 127.8.204.189:45951
I20260927 22:53:47.282014 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.236852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.8.204.130:52745)
0927 22:53:47.252402 (+ 15550us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.252446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.255127 (+ 2681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.261293 (+ 6166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.261298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.261701 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:47.261985 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.262030 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.264101 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.264107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.265839 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.265856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.267212 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.267219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.267453 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.268085 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:47.268113 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:47.274047 (+ 5934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:47.278555 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.278559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.278562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.279075 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.279490 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.279495 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.279498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.279663 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.280972 (+ 1309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:47.280977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:47.281149 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.281283 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.281558 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15234,"mutex_wait_us":50,"thread_start_us":22447,"threads_started":1}
I20260927 22:53:47.282258 16267 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:47.282907 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.238349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:52745 (local address 127.8.204.188:45411)
0927 22:53:47.255968 (+ 17619us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.255977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.256001 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:47.256068 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.256076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.256176 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.256300 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.262192 (+ 5892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.263353 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.266020 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.266803 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.266912 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.274266 (+ 7354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.274273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.274277 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.274318 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.278402 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.279205 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.279211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.279217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.279365 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.279805 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.279810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.279813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.280151 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.280416 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.282331 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.282535 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17450,"thread_start_us":17281,"threads_started":1}
I20260927 22:53:47.283159 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.238683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:33095 (local address 127.8.204.190:44551)
0927 22:53:47.257506 (+ 18823us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.257516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.257540 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:47.257623 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.257630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.257728 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.257853 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.261280 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.262395 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.265126 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.266237 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.266363 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.271075 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.271083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.271107 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.271154 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.275388 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.278763 (+ 3375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.278769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.278774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.278931 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.279463 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.279469 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.279472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.279818 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.279934 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.281422 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.282681 (+ 1259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18667,"thread_start_us":18554,"threads_started":1}
I20260927 22:53:47.283165 16267 heartbeater.cc:511] Master 127.8.204.189:45951 requested a full tablet report, sending...
I20260927 22:53:47.283699 16265 heartbeater.cc:347] Connected to a master server at 127.8.204.188:45411
I20260927 22:53:47.283843 16265 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:47.284117 16266 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44551
I20260927 22:53:47.284248 16266 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:47.284324 16265 heartbeater.cc:511] Master 127.8.204.188:45411 requested a full tablet report, sending...
I20260927 22:53:47.284741 16266 heartbeater.cc:511] Master 127.8.204.190:44551 requested a full tablet report, sending...
I20260927 22:53:47.285044 15752 ts_manager.cc:194] Registered new tserver with Master: 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231)
I20260927 22:53:47.285424 15940 ts_manager.cc:194] Registered new tserver with Master: 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231)
I20260927 22:53:47.285774 15154 ts_manager.cc:194] Re-registered known tserver with Master: 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231)
I20260927 22:53:47.286576 15154 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:33095
I20260927 22:53:47.287055 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.243499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.8.204.131:44645)
0927 22:53:47.251840 (+ 8341us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.251850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.255895 (+ 4045us) server_negotiation.cc:407] Connection header received
0927 22:53:47.256051 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.256060 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.256298 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.256434 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.259049 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.260477 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.262418 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.263311 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.263445 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.273512 (+ 10067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.273530 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.273550 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.273652 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.277890 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.279129 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.279138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.279142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.279323 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.280240 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.280246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.280249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.280601 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.280893 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.283961 (+ 3068us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.284246 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8149,"thread_start_us":8001,"threads_started":1}
I20260927 22:53:47.288281 15606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:47.370190 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15483
I20260927 22:53:47.386217 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:44645
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:44551
--webserver_port=43493
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:53:47.687393 16118 heartbeater.cc:503] Master 127.8.204.190:44551 was elected leader, sending a full tablet report...
W20260927 22:53:47.706779 16278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:47.707032 16278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:47.707130 16278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:47.714198 16278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:47.714299 16278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:47.714409 16278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:47.723822 16278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:44645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=43493
--tserver_master_addrs=127.8.204.190:44551,127.8.204.189:45951,127.8.204.188:45411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:47.724997 16278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:47.726197 16278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:47.738919 16286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:47.739082 16284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:47.738897 16283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:47.743227 16278 server_base.cc:1034] running on GCE node
I20260927 22:53:47.743929 16278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:47.744889 16278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:47.746140 16278 hybrid_clock.cc:648] HybridClock initialized: now 1790549627746093 us; error 56 us; skew 500 ppm
I20260927 22:53:47.746483 16278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:47.749637 16278 init.cc:377] Logged in from keytab as oryx/127.8.204.131@KRBTEST.COM (short username oryx)
I20260927 22:53:47.751151 16278 webserver.cc:460] Webserver started at http://127.8.204.131:43493/ using document root <none> and password file <none>
I20260927 22:53:47.751933 16278 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:47.752094 16278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:47.757421 16278 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:53:47.760241 16293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:47.761703 16278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:53:47.761873 16278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec"
format_stamp: "Formatted at 2026-09-27 22:53:44 on dist-test-slave-40vx"
server_key: "ddd1d7cdcff0c1b428a3af242d0b783e"
server_key_iv: "daafc43ae143303e66b18006984b675a"
server_key_version: "encryptionkey@0"
I20260927 22:53:47.762969 16278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:47.776015 16278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:47.779918 16278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:47.780200 16278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:47.781086 16278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:53:47.782546 16278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:47.782663 16278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:47.782789 16278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:47.782884 16278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:47.822963 16278 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:44645
I20260927 22:53:47.823014 16405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:44645 every 8 connection(s)
I20260927 22:53:47.824375 16278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:47.825618 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 16278
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.188@KRBTEST.COM
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
Sep 27 22:53:47 dist-test-slave-40vx krb5kdc[15110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549627, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.189@KRBTEST.COM
I20260927 22:53:47.875155 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.828879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.8.204.131:33747)
0927 22:53:47.845148 (+ 16269us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.845166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.845287 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.851429 (+ 6142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.851433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.852020 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:47.852287 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.852298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.854009 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.854017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.855768 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.855786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.857076 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.857085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.857468 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.858384 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:47.858424 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:47.862616 (+ 4192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:47.868663 (+ 6047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.868676 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.868704 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.869575 (+ 871us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.871616 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.871623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.871629 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.871894 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.872556 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:47.872565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:47.873002 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.873320 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.873817 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16146,"thread_start_us":15992,"threads_started":1}
I20260927 22:53:47.876658 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.827442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:33747 (local address 127.8.204.188:45411)
0927 22:53:47.850106 (+ 22664us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.850113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.850133 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:53:47.850194 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.850202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.850307 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.852446 (+ 2139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.852626 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.853829 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.855985 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.856816 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.856959 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.862943 (+ 5984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.862952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.862960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.863010 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.868468 (+ 5458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.871330 (+ 2862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.871339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.871343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.871474 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.872065 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.872071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.872078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.872413 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.875915 (+ 3502us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.876048 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.876275 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22510,"thread_start_us":22368,"threads_started":1}
I20260927 22:53:47.877872 16406 heartbeater.cc:347] Connected to a master server at 127.8.204.188:45411
I20260927 22:53:47.878178 16406 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:47.879132 16406 heartbeater.cc:511] Master 127.8.204.188:45411 requested a full tablet report, sending...
I20260927 22:53:47.880141 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.828166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:33741)
0927 22:53:47.840364 (+ 12198us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.840386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.844441 (+ 4055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.845324 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.845338 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.846380 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:47.848145 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.848180 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.850481 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.850489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.853493 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.853504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.859480 (+ 5976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.859489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.859722 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.862472 (+ 2750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:47.862508 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:47.870326 (+ 7818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:47.875902 (+ 5576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.875907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.875910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.876397 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.877241 (+ 844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.877245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.877249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.877390 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.879095 (+ 1705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:47.879101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:47.879255 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.879407 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.879665 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12010,"mutex_wait_us":341,"thread_start_us":8338,"threads_started":1}
I20260927 22:53:47.881094 16413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.828412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:33741 (local address 127.8.204.190:44551)
0927 22:53:47.836703 (+ 8291us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.836714 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.840549 (+ 3835us) server_negotiation.cc:407] Connection header received
0927 22:53:47.844598 (+ 4049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.844606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.844726 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.844837 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.848376 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.849866 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.853629 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.855898 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.856056 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.870761 (+ 14705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.870771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.870779 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.870836 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.874682 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.876913 (+ 2231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.876920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.876925 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.877086 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.877714 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.877720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.877723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.878082 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.878226 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.880450 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.880697 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8139,"thread_start_us":8041,"threads_started":1}
I20260927 22:53:47.881333 15940 ts_manager.cc:194] Registered new tserver with Master: 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645)
I20260927 22:53:47.881826 16407 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44551
I20260927 22:53:47.881939 16407 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:47.882359 16407 heartbeater.cc:511] Master 127.8.204.190:44551 requested a full tablet report, sending...
I20260927 22:53:47.883803 15154 ts_manager.cc:194] Re-registered known tserver with Master: 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645)
I20260927 22:53:47.885030 15154 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:33741
I20260927 22:53:47.888414 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.828755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:52145 (local address 127.8.204.189:45951)
0927 22:53:47.847015 (+ 18260us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.847022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.847046 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:47.847106 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.847111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.847189 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.847345 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.850064 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.850951 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.864173 (+ 13222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.865039 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.865448 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.879109 (+ 13661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.879116 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.879121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.879166 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.882620 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.884407 (+ 1787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.884414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.884419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.884637 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.886649 (+ 2012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.886655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.886660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.887051 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.887203 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.887834 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.888038 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18138,"thread_start_us":18794,"threads_started":1}
I20260927 22:53:47.888414 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.828166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.8.204.131:52145)
0927 22:53:47.837388 (+ 9222us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.837455 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.844052 (+ 6597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.848403 (+ 4351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.848410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.848980 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:47.849296 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.849310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.851103 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.851108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.861185 (+ 10077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.861203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.865783 (+ 4580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.865791 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.866050 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.866929 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:47.866958 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:47.878885 (+ 11927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:47.883137 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.883142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.883145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.884276 (+ 1131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.886316 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:47.886321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:47.886324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:47.886514 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:47.887414 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:47.887419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:47.887557 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.887690 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.888016 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8878,"thread_start_us":11737,"threads_started":1}
I20260927 22:53:47.889581 16408 heartbeater.cc:347] Connected to a master server at 127.8.204.189:45951
I20260927 22:53:47.889709 16408 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:47.890116 16408 heartbeater.cc:511] Master 127.8.204.189:45951 requested a full tablet report, sending...
I20260927 22:53:47.891302 15752 ts_manager.cc:194] Registered new tserver with Master: 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645)
I20260927 22:53:47.894196 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.852931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42738 (local address 127.8.204.188:45411)
0927 22:53:47.853404 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.853410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.870084 (+ 16674us) server_negotiation.cc:407] Connection header received
0927 22:53:47.870974 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.870982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.871090 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.871172 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.874095 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.875724 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.879068 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.879718 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.879817 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.884737 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.884744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.884749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.884784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.889633 (+ 4849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.891392 (+ 1759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.891398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.891402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.891550 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.892061 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.892066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.892069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.892408 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.892689 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.893624 (+ 935us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.893880 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":235,"threads_started":1}
I20260927 22:53:47.896852 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.852717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.8.204.189:45951)
0927 22:53:47.869690 (+ 16973us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.869697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.869717 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:53:47.869769 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.869772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.869839 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.873876 (+ 4037us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.874095 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.875749 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.884495 (+ 8746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.886197 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.887167 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.889874 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.889882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.889889 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.889929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.893862 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.894727 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.894731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.894735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.894889 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.895368 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.895373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.895375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.895717 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.895957 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.896282 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.896472 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16826,"thread_start_us":16715,"threads_started":1}
I20260927 22:53:47.897446 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.853987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.8.204.190:44551)
0927 22:53:47.871378 (+ 17391us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.871384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.871405 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:47.871454 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.871458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.871540 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.871643 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:47.878619 (+ 6976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.882259 (+ 3640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.885198 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.885907 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.886049 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.891310 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:47.891317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:47.891324 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:47.891364 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:47.894817 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.895513 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.895518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.895522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.895650 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:47.896039 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:47.896043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:47.896046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:47.896281 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:47.896516 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.896870 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.897071 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17224,"thread_start_us":17115,"threads_started":1}
I20260927 22:53:47.901181 15154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:47.908052 15154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:47.949725 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.926311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:44645 (local address 127.0.0.1:57226)
0927 22:53:47.927324 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.927344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.927468 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.943078 (+ 15610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.943086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.943124 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:47.943561 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.943572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.945356 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.945361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.946860 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.946872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.948838 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.948844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.949033 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.949067 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.949127 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.949260 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":1715,"threads_started":1}
I20260927 22:53:47.950736 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.927023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.8.204.131:44645)
0927 22:53:47.942340 (+ 15317us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.942352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.942391 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:53:47.942508 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.942519 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.942840 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.943051 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:47.943733 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.945178 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.947022 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.948626 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.949972 (+ 1346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.950183 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.950258 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.950408 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13048,"thread_start_us":14998,"threads_started":1}
I20260927 22:53:47.952549 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.928463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.8.204.130:42231)
0927 22:53:47.943738 (+ 15275us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.943750 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.943781 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:53:47.943881 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.943891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.944160 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.944353 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:47.945112 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.947073 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.949804 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.951182 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.951311 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.951495 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.952124 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.952279 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15083,"thread_start_us":14977,"threads_started":1}
I20260927 22:53:47.952628 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.928202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42231 (local address 127.0.0.1:59134)
0927 22:53:47.941766 (+ 13564us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.941789 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.941924 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.944361 (+ 2437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.944365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.944388 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:47.944686 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.944696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.947222 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.947228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.949625 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.949637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.951662 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.951668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.951867 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.951968 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.952090 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.952214 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13411,"mutex_wait_us":30,"thread_start_us":13246,"threads_started":1}
I20260927 22:53:47.954648 16341 tablet_service.cc:1467] Processing CreateTablet for tablet 86415ccae85e43b39e5f7feec7612474 (DEFAULT_TABLE table=test-table [id=1e1210781a8f42868f4ae094db789548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:47.955323 16200 tablet_service.cc:1467] Processing CreateTablet for tablet 86415ccae85e43b39e5f7feec7612474 (DEFAULT_TABLE table=test-table [id=1e1210781a8f42868f4ae094db789548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:47.955994 16341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86415ccae85e43b39e5f7feec7612474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:47.956466 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86415ccae85e43b39e5f7feec7612474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:47.962215 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.928996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.8.204.129:41189)
0927 22:53:47.947723 (+ 18727us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:47.947733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:47.948345 (+ 612us) server_negotiation.cc:407] Connection header received
0927 22:53:47.948506 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:47.948513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:47.948698 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:47.948875 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:47.950434 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.951719 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.959150 (+ 7431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.960696 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.961231 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.961421 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:47.961765 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 22:53:47.961933 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18575,"thread_start_us":18449,"threads_started":1}
I20260927 22:53:47.962728 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:47.928412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41189 (local address 127.0.0.1:51504)
0927 22:53:47.948232 (+ 19820us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:47.948246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:47.948380 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:47.948863 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:47.948868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:47.948892 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:47.950248 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.950260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.951876 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.951881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.954160 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:47.954174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:47.961315 (+ 7141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:47.961322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:47.961521 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:47.961556 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:53:47.961667 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:53:47.961862 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19716,"mutex_wait_us":49,"thread_start_us":19536,"threads_started":1}
I20260927 22:53:47.966369 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 86415ccae85e43b39e5f7feec7612474 (DEFAULT_TABLE table=test-table [id=1e1210781a8f42868f4ae094db789548]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:47.967760 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86415ccae85e43b39e5f7feec7612474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:47.969637 16441 tablet_bootstrap.cc:492] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Bootstrap starting.
I20260927 22:53:47.971657 16442 tablet_bootstrap.cc:492] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Bootstrap starting.
I20260927 22:53:47.974555 16441 tablet_bootstrap.cc:654] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:47.974830 16442 tablet_bootstrap.cc:654] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:47.975802 16441 log.cc:824] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:47.975893 16442 log.cc:824] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:47.978789 16441 tablet_bootstrap.cc:492] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: No bootstrap required, opened a new log
I20260927 22:53:47.979049 16441 ts_tablet_manager.cc:1397] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260927 22:53:47.979946 16442 tablet_bootstrap.cc:492] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: No bootstrap required, opened a new log
I20260927 22:53:47.980207 16442 ts_tablet_manager.cc:1397] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 22:53:47.981755 16445 tablet_bootstrap.cc:492] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Bootstrap starting.
I20260927 22:53:47.984809 16445 tablet_bootstrap.cc:654] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:47.986181 16445 log.cc:824] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:47.988487 16445 tablet_bootstrap.cc:492] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: No bootstrap required, opened a new log
I20260927 22:53:47.988744 16445 ts_tablet_manager.cc:1397] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:53:47.990535 16442 raft_consensus.cc:348] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:47.990947 16442 raft_consensus.cc:374] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:47.991043 16442 raft_consensus.cc:729] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0227056099ad4354bde4a3cae8940c96, State: Initialized, Role: FOLLOWER
I20260927 22:53:47.991189 16441 raft_consensus.cc:348] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:47.991583 16441 raft_consensus.cc:374] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:47.991786 16441 raft_consensus.cc:729] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f20ca8f443449d4b8f2075e3ad3b8ec, State: Initialized, Role: FOLLOWER
I20260927 22:53:47.991691 16442 consensus_queue.cc:260] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:47.992905 16441 consensus_queue.cc:260] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:47.995225 16441 ts_tablet_manager.cc:1428] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260927 22:53:47.997093 16407 heartbeater.cc:503] Master 127.8.204.190:44551 was elected leader, sending a full tablet report...
I20260927 22:53:47.997684 16445 raft_consensus.cc:348] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:47.998076 16445 raft_consensus.cc:374] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:47.998164 16445 raft_consensus.cc:729] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fd2a85f6534816b8a77be4c332c34a, State: Initialized, Role: FOLLOWER
I20260927 22:53:47.998792 16445 consensus_queue.cc:260] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:48.000500 16442 ts_tablet_manager.cc:1428] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Time spent starting tablet: real 0.020s user 0.010s sys 0.010s
I20260927 22:53:48.000754 16266 heartbeater.cc:503] Master 127.8.204.190:44551 was elected leader, sending a full tablet report...
I20260927 22:53:48.006106 16445 ts_tablet_manager.cc:1428] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
I20260927 22:53:48.028496 16448 raft_consensus.cc:482] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:48.028810 16448 raft_consensus.cc:504] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:48.030511 16448 leader_election.cc:288] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189), 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231)
I20260927 22:53:48.035995 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.030941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:59843 (local address 127.8.204.130:42231)
0927 22:53:48.031426 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.031433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.031471 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:53:48.031535 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.031539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.031622 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.031713 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:48.032096 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.032893 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.034384 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.035131 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.035322 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.035393 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.035650 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.035765 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260927 22:53:48.036031 16412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.030740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42231 (local address 127.8.204.131:59843)
0927 22:53:48.031159 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:48.031177 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:48.031310 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:48.031755 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:48.031760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:48.031785 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:48.031941 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:48.031954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.033031 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.033036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:48.034242 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:48.034252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.035322 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.035328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:48.035497 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.035530 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:53:48.035629 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:53:48.035759 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":170}
I20260927 22:53:48.036231 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.030774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38097 (local address 127.8.204.129:41189)
0927 22:53:48.031178 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.031191 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.031208 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:53:48.031320 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.031325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.031394 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.031490 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:48.032330 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.033097 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.034545 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.035396 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.035553 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.035631 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.035877 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.035988 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260927 22:53:48.036430 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.030717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41189 (local address 127.8.204.131:38097)
0927 22:53:48.030908 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:48.030927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:48.031044 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:48.031519 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:48.031524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:48.031549 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:48.031737 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:48.031748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.033235 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.033241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:48.034398 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:48.034408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.035548 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.035554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:48.035736 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.035758 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:53:48.035854 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:53:48.035977 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:53:48.036931 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86415ccae85e43b39e5f7feec7612474" candidate_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0227056099ad4354bde4a3cae8940c96" is_pre_election: true
I20260927 22:53:48.037388 16220 raft_consensus.cc:2454] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f20ca8f443449d4b8f2075e3ad3b8ec in term 0.
I20260927 22:53:48.037428 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86415ccae85e43b39e5f7feec7612474" candidate_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58fd2a85f6534816b8a77be4c332c34a" is_pre_election: true
I20260927 22:53:48.037909 16072 raft_consensus.cc:2454] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f20ca8f443449d4b8f2075e3ad3b8ec in term 0.
I20260927 22:53:48.038012 16296 leader_election.cc:302] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0227056099ad4354bde4a3cae8940c96, 2f20ca8f443449d4b8f2075e3ad3b8ec; no voters:
I20260927 22:53:48.038375 16448 raft_consensus.cc:2781] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:48.038506 16448 raft_consensus.cc:482] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:48.038609 16448 raft_consensus.cc:3037] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:48.040779 16448 raft_consensus.cc:504] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:48.041323 16448 leader_election.cc:288] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [CANDIDATE]: Term 1 election: Requested vote from peers 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189), 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231)
I20260927 22:53:48.041702 16072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86415ccae85e43b39e5f7feec7612474" candidate_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58fd2a85f6534816b8a77be4c332c34a"
I20260927 22:53:48.041735 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86415ccae85e43b39e5f7feec7612474" candidate_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0227056099ad4354bde4a3cae8940c96"
I20260927 22:53:48.041888 16072 raft_consensus.cc:3037] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:48.041893 16220 raft_consensus.cc:3037] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:48.045365 16220 raft_consensus.cc:2454] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f20ca8f443449d4b8f2075e3ad3b8ec in term 1.
I20260927 22:53:48.045367 16072 raft_consensus.cc:2454] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f20ca8f443449d4b8f2075e3ad3b8ec in term 1.
I20260927 22:53:48.045800 16296 leader_election.cc:302] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0227056099ad4354bde4a3cae8940c96, 2f20ca8f443449d4b8f2075e3ad3b8ec; no voters:
I20260927 22:53:48.046049 16448 raft_consensus.cc:2781] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:48.046491 16448 raft_consensus.cc:686] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 LEADER]: Becoming Leader. State: Replica: 2f20ca8f443449d4b8f2075e3ad3b8ec, State: Running, Role: LEADER
I20260927 22:53:48.046977 16448 consensus_queue.cc:237] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } }
I20260927 22:53:48.051812 15155 catalog_manager.cc:5495] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131). New cstate: current_term: 1 leader_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } health_report { overall_health: UNKNOWN } } }
W20260927 22:53:48.098425 16409 tablet.cc:2369] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:48.098721 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.093588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.8.204.131:44645)
0927 22:53:48.093791 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.093796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.093816 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:53:48.094037 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.094040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.094117 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.094244 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:48.094543 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.095230 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.096530 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.096911 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.097006 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.097420 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:48.097913 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:48.098023 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.098142 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.098204 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:48.106350 16220 raft_consensus.cc:1261] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Refusing update from remote peer 2f20ca8f443449d4b8f2075e3ad3b8ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:48.106369 16072 raft_consensus.cc:1261] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Refusing update from remote peer 2f20ca8f443449d4b8f2075e3ad3b8ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:48.107343 16450 consensus_queue.cc:1035] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:48.107678 16448 consensus_queue.cc:1035] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:48.139684 15155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:48.139936 15155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:48.146292 15155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36a20dcf2145413292db3c418e7e97ed: Sending DeleteTablet for 3 replicas of tablet 86415ccae85e43b39e5f7feec7612474
I20260927 22:53:48.147447 16341 tablet_service.cc:1514] Processing DeleteTablet for tablet 86415ccae85e43b39e5f7feec7612474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:48 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:57226
I20260927 22:53:48.147606 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 86415ccae85e43b39e5f7feec7612474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:48 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:51504
I20260927 22:53:48.149451 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet 86415ccae85e43b39e5f7feec7612474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:48 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:59134
I20260927 22:53:48.152192 16466 tablet_replica.cc:330] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: stopping tablet replica
I20260927 22:53:48.152709 16466 raft_consensus.cc:2229] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:48.153276 16466 raft_consensus.cc:2258] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:48.155745 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15123
I20260927 22:53:48.157138 15185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:48.158968 16464 tablet_replica.cc:330] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: stopping tablet replica
I20260927 22:53:48.159490 16464 raft_consensus.cc:2229] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:48.160614 16464 raft_consensus.cc:2258] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:48.160670 16466 ts_tablet_manager.cc:1905] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:48.167088 16463 tablet_replica.cc:330] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: stopping tablet replica
I20260927 22:53:48.167618 16464 ts_tablet_manager.cc:1905] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:48.168402 16466 ts_tablet_manager.cc:1918] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:48.168617 16466 log.cc:1196] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/86415ccae85e43b39e5f7feec7612474
I20260927 22:53:48.169209 16466 ts_tablet_manager.cc:1939] T 86415ccae85e43b39e5f7feec7612474 P 0227056099ad4354bde4a3cae8940c96: Deleting consensus metadata
I20260927 22:53:48.173148 16463 raft_consensus.cc:2229] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:48.173900 16463 raft_consensus.cc:2258] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:48.177043 16463 ts_tablet_manager.cc:1905] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:48.179488 16464 ts_tablet_manager.cc:1918] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:48.179659 16464 log.cc:1196] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/86415ccae85e43b39e5f7feec7612474
I20260927 22:53:48.180195 16464 ts_tablet_manager.cc:1939] T 86415ccae85e43b39e5f7feec7612474 P 58fd2a85f6534816b8a77be4c332c34a: Deleting consensus metadata
I20260927 22:53:48.187816 16463 ts_tablet_manager.cc:1918] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:48.188000 16463 log.cc:1196] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/86415ccae85e43b39e5f7feec7612474
I20260927 22:53:48.188647 16463 ts_tablet_manager.cc:1939] T 86415ccae85e43b39e5f7feec7612474 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Deleting consensus metadata
I20260927 22:53:48.189996 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.150249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44480 (local address 127.8.204.189:45951)
0927 22:53:48.150559 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.150568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.154187 (+ 3619us) server_negotiation.cc:407] Connection header received
0927 22:53:48.154464 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.154473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.154599 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.154704 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:48.161021 (+ 6317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.162487 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.170956 (+ 8469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.171821 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.171982 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.182323 (+ 10341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:48.182331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:48.182336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:48.182398 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:48.186468 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.187400 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.187406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.187410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.187565 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.188140 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.188146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.188149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.188502 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:48.188805 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.189404 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.189646 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 22:53:48.207679 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.154201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.8.204.188:45411)
0927 22:53:48.154380 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.154387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.191260 (+ 36873us) server_negotiation.cc:407] Connection header received
0927 22:53:48.191542 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.191548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.191666 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.191738 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:48.193827 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.195376 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.197194 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.197848 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.197980 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.201273 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:48.201280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:48.201285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:48.201324 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:48.204801 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.205535 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.205541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.205544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.205686 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.206065 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.206070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.206075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.206386 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:48.206747 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.207096 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.207333 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:53:48.327970 15139 catalog_manager.cc:4841] TS 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645): tablet 86415ccae85e43b39e5f7feec7612474 (table test-table [id=1e1210781a8f42868f4ae094db789548]) successfully deleted
I20260927 22:53:48.328379 15139 catalog_manager.cc:4841] TS 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189): tablet 86415ccae85e43b39e5f7feec7612474 (table test-table [id=1e1210781a8f42868f4ae094db789548]) successfully deleted
I20260927 22:53:48.328416 15141 catalog_manager.cc:4841] TS 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231): tablet 86415ccae85e43b39e5f7feec7612474 (table test-table [id=1e1210781a8f42868f4ae094db789548]) successfully deleted
W20260927 22:53:48.328617 15141 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.204.130:42231, user_credentials={real_user=slave}} blocked reactor thread for 157133us
I20260927 22:53:48.328840 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15123
I20260927 22:53:48.392174 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.363471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.8.204.188:45411)
0927 22:53:48.363716 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.363725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.375030 (+ 11305us) server_negotiation.cc:407] Connection header received
0927 22:53:48.376076 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.376082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.376188 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.376259 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:48.378531 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.379895 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.381196 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.381773 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.381914 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.385498 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:48.385505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:48.385509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:48.385560 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:48.389022 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.389810 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.389818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.389824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.389963 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.390363 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.390371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.390377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.390763 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:48.391001 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.391418 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.391655 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:53:48.393941 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:48.363880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.8.204.189:45951)
0927 22:53:48.364050 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:48.364056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:48.374982 (+ 10926us) server_negotiation.cc:407] Connection header received
0927 22:53:48.375180 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:48.375186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:48.375305 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:48.375406 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:48.381434 (+ 6028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.382372 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.384034 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:48.384432 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:48.384571 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:48.387593 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:48.387598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:48.387602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:48.387634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:48.390592 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.391397 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.391403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.391406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.391540 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:48.392027 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:48.392034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:48.392037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:48.392461 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:48.392789 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:48.393300 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 22:53:48.393594 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:53:49.126202 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.125274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:39935)
0927 22:53:49.125863 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.126024 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":310,"threads_started":1}
W20260927 22:53:49.126652 16118 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:44551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.132261 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.131432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:56649)
0927 22:53:49.131889 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.132074 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
W20260927 22:53:49.132647 16407 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:44551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.134933 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.134231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:58533)
0927 22:53:49.134667 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.134787 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":251,"threads_started":1}
W20260927 22:53:49.135383 16266 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:44551 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.646118 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a20dcf2145413292db3c418e7e97ed)
I20260927 22:53:49.646356 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } }
I20260927 22:53:49.647229 16479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36a20dcf2145413292db3c418e7e97ed)
I20260927 22:53:49.647441 16479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } }
I20260927 22:53:49.647898 16478 leader_election.cc:288] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551), 79389ed8503b43068a63586589802748 (127.8.204.188:45411)
I20260927 22:53:49.648325 16479 leader_election.cc:288] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551), d0c24fd1fec242dfaab8755016f0d8da (127.8.204.189:45951)
I20260927 22:53:49.649376 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.648162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41360)
0927 22:53:49.648853 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.649050 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":449,"threads_started":1}
W20260927 22:53:49.649711 15739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.659462 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.648380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41362)
0927 22:53:49.658332 (+ 9952us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.659117 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9759,"thread_start_us":9621,"threads_started":1}
I20260927 22:53:49.659740 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.648940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45411 (local address 127.0.0.1:50198)
0927 22:53:49.649576 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.649592 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:49.649738 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:49.650676 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:49.650684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:49.650721 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:49.651051 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.651066 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.653210 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.653218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.655841 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.655861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.658636 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.658645 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.658857 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.658908 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:53:49.659043 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:53:49.659245 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":9771,"threads_started":1}
W20260927 22:53:49.659713 15927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.660552 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.648363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.8.204.188:45411)
0927 22:53:49.650278 (+ 1915us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:49.650287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:49.650311 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:53:49.650394 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:49.650401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:49.650496 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:49.651224 (+ 728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:49.651495 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.653013 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.656117 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.657768 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.659339 (+ 1571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.659445 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:49.659512 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 22:53:49.659640 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1764,"thread_start_us":1608,"threads_started":1}
I20260927 22:53:49.661178 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0c24fd1fec242dfaab8755016f0d8da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "79389ed8503b43068a63586589802748" is_pre_election: true
I20260927 22:53:49.661345 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.660968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41364)
0927 22:53:49.661098 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.661184 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 22:53:49.661600 15950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0c24fd1fec242dfaab8755016f0d8da in term 1.
W20260927 22:53:49.661751 15739 leader_election.cc:334] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551): Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.662199 15738 leader_election.cc:302] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79389ed8503b43068a63586589802748, d0c24fd1fec242dfaab8755016f0d8da; no voters: 36a20dcf2145413292db3c418e7e97ed
I20260927 22:53:49.662595 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.662189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41370)
0927 22:53:49.662369 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.662453 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:53:49.662729 16478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:53:49.662874 16478 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36a20dcf2145413292db3c418e7e97ed)
I20260927 22:53:49.663039 16478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 1 FOLLOWER]: Advancing to term 2
W20260927 22:53:49.662937 15927 leader_election.cc:334] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551): Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.666247 16478 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } }
I20260927 22:53:49.667057 16478 leader_election.cc:288] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [CANDIDATE]: Term 2 election: Requested vote from peers 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551), 79389ed8503b43068a63586589802748 (127.8.204.188:45411)
I20260927 22:53:49.667625 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.667192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41386)
0927 22:53:49.667380 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.667489 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 22:53:49.667678 15950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0c24fd1fec242dfaab8755016f0d8da" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "79389ed8503b43068a63586589802748"
I20260927 22:53:49.667901 15950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:53:49.668686 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.668268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41396)
0927 22:53:49.668417 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.668497 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 22:53:49.668921 15739 leader_election.cc:334] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551): Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
I20260927 22:53:49.669118 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.650341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:45951 (local address 127.0.0.1:41880)
0927 22:53:49.658709 (+ 8368us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.658731 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:49.658852 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:49.661233 (+ 2381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:49.661238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:49.661261 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:49.661650 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.661663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.663863 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.663869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.665823 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.665838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.668265 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.668271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.668462 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.668504 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:53:49.668611 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:53:49.668766 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8200,"thread_start_us":8050,"threads_started":1}
I20260927 22:53:49.669157 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.660200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41880 (local address 127.8.204.189:45951)
0927 22:53:49.660926 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:49.660931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:49.660953 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:53:49.661006 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:49.661010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:49.661076 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:49.661833 (+ 757us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:49.662408 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.663721 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.666034 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.667494 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.668217 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.668352 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:49.668652 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 22:53:49.668783 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":455,"threads_started":1}
I20260927 22:53:49.669744 15762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "79389ed8503b43068a63586589802748" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d0c24fd1fec242dfaab8755016f0d8da" is_pre_election: true
I20260927 22:53:49.670006 15762 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79389ed8503b43068a63586589802748 in current term 2: Already voted for candidate d0c24fd1fec242dfaab8755016f0d8da in this term.
I20260927 22:53:49.670495 15924 leader_election.cc:302] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79389ed8503b43068a63586589802748; no voters: 36a20dcf2145413292db3c418e7e97ed, d0c24fd1fec242dfaab8755016f0d8da
I20260927 22:53:49.670784 15950 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0c24fd1fec242dfaab8755016f0d8da in term 2.
I20260927 22:53:49.670917 16479 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:53:49.671203 15738 leader_election.cc:302] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79389ed8503b43068a63586589802748, d0c24fd1fec242dfaab8755016f0d8da; no voters: 36a20dcf2145413292db3c418e7e97ed
I20260927 22:53:49.671500 16478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:53:49.672036 16478 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [term 2 LEADER]: Becoming Leader. State: Replica: d0c24fd1fec242dfaab8755016f0d8da, State: Running, Role: LEADER
I20260927 22:53:49.672441 16478 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } }
I20260927 22:53:49.674232 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c24fd1fec242dfaab8755016f0d8da. Latest consensus state: current_term: 2 leader_uuid: "d0c24fd1fec242dfaab8755016f0d8da" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:49.674520 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:49.675132 16490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:49.678700 16490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:49.679558 16490 catalog_manager.cc:1259] Loaded cluster ID: d55fb42e8da6423e8e026d2983783b7a
I20260927 22:53:49.679629 16490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:49.680701 16490 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:49.681522 16490 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Loaded TSK: 0
I20260927 22:53:49.682480 16490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:53:49.720791 15751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:49.727801 15751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:49.731730 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.731332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41414)
0927 22:53:49.731496 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.731599 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 22:53:49.731974 15950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [term 2 FOLLOWER]: Refusing update from remote peer d0c24fd1fec242dfaab8755016f0d8da: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 22:53:49.732825 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.732347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41422)
0927 22:53:49.732565 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.732659 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:53:49.732720 16488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [LEADER]: Connected to new peer: Peer: permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 22:53:49.733026 15739 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da -> Peer 36a20dcf2145413292db3c418e7e97ed (127.8.204.190:44551): Couldn't send request to peer 36a20dcf2145413292db3c418e7e97ed. Status: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:53:49.735554 16479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0c24fd1fec242dfaab8755016f0d8da. Latest consensus state: current_term: 2 leader_uuid: "d0c24fd1fec242dfaab8755016f0d8da" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:49.735773 16479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:53:49.743537 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d0c24fd1fec242dfaab8755016f0d8da" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:49.743863 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:49.743780 16479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d0c24fd1fec242dfaab8755016f0d8da" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36a20dcf2145413292db3c418e7e97ed" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44551 } } peers { permanent_uuid: "d0c24fd1fec242dfaab8755016f0d8da" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 45951 } attrs { promote: false } } peers { permanent_uuid: "79389ed8503b43068a63586589802748" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45411 } attrs { promote: false } } }
I20260927 22:53:49.744033 16479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79389ed8503b43068a63586589802748 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:53:49.758899 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.751784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41189 (local address 127.0.0.1:46692)
0927 22:53:49.751932 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.751952 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:49.752098 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:49.752658 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:49.752663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:49.752689 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:49.752922 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.752934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.755379 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.755385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.756531 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.756541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.758107 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.758113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.758293 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.758355 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 22:53:49.758438 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 22:53:49.758582 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 22:53:49.758916 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.751869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46692 (local address 127.8.204.129:41189)
0927 22:53:49.752332 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:49.752339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:49.752358 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:53:49.752427 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:49.752432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:49.752504 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:49.753972 (+ 1468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:49.754151 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.755237 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.756682 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.757936 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.758042 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.758157 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:49.758479 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 22:53:49.758610 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":250,"threads_started":1}
I20260927 22:53:49.759927 16052 tablet_service.cc:1467] Processing CreateTablet for tablet 9aabef284bbd4510b0aecab0988153d1 (DEFAULT_TABLE table=test-table [id=4c8ac03d6aa547dd8810df7d05c2c807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:49.760418 16052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aabef284bbd4510b0aecab0988153d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:49.761797 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.753255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.8.204.131:44645)
0927 22:53:49.753882 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:49.753887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:49.753907 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:53:49.753973 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:49.753977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:49.754061 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:49.754156 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:49.755193 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.756637 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.759403 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.760731 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.760847 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.761004 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:49.761297 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:53:49.761435 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":426,"threads_started":1}
I20260927 22:53:49.761802 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.752995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:44645 (local address 127.0.0.1:60504)
0927 22:53:49.753638 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.753659 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:49.753775 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:49.754332 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:49.754336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:49.754707 (+ 371us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:49.754989 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.755000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.756821 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.756826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.758618 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.758629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.760931 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.760940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.761131 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.761165 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:53:49.761263 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:53:49.761389 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":1,"thread_start_us":306,"threads_started":1}
I20260927 22:53:49.762516 16341 tablet_service.cc:1467] Processing CreateTablet for tablet 9aabef284bbd4510b0aecab0988153d1 (DEFAULT_TABLE table=test-table [id=4c8ac03d6aa547dd8810df7d05c2c807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:49.762929 16341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aabef284bbd4510b0aecab0988153d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:49.764053 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.752620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42231 (local address 127.0.0.1:40518)
0927 22:53:49.752882 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.752904 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:49.753063 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:49.757289 (+ 4226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:49.757294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:49.757317 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:49.757774 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.757786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.759396 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.759401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.761414 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:49.761426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.763359 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.763368 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:49.763545 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.763578 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:53:49.763653 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:53:49.763774 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 22:53:49.764884 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.752725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.8.204.130:42231)
0927 22:53:49.756979 (+ 4254us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:49.756986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:49.757004 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:53:49.757066 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:49.757072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:49.757142 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:49.757936 (+ 794us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:49.758081 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.759239 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.761613 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:49.763098 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:49.764336 (+ 1238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:49.764430 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:49.764475 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 22:53:49.764611 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4145,"thread_start_us":4058,"threads_started":1}
I20260927 22:53:49.766280 16500 tablet_bootstrap.cc:492] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Bootstrap starting.
I20260927 22:53:49.766497 16200 tablet_service.cc:1467] Processing CreateTablet for tablet 9aabef284bbd4510b0aecab0988153d1 (DEFAULT_TABLE table=test-table [id=4c8ac03d6aa547dd8810df7d05c2c807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:49.766911 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aabef284bbd4510b0aecab0988153d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:49.768153 16500 tablet_bootstrap.cc:654] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:49.769982 16500 tablet_bootstrap.cc:492] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: No bootstrap required, opened a new log
I20260927 22:53:49.770150 16500 ts_tablet_manager.cc:1397] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 22:53:49.770923 16500 raft_consensus.cc:348] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:49.771129 16500 raft_consensus.cc:374] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:49.771221 16500 raft_consensus.cc:729] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fd2a85f6534816b8a77be4c332c34a, State: Initialized, Role: FOLLOWER
I20260927 22:53:49.771486 16500 consensus_queue.cc:260] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:49.775259 16504 tablet_bootstrap.cc:492] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Bootstrap starting.
I20260927 22:53:49.775835 16501 tablet_bootstrap.cc:492] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Bootstrap starting.
I20260927 22:53:49.777562 16501 tablet_bootstrap.cc:654] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:49.777760 16504 tablet_bootstrap.cc:654] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:49.779453 16500 ts_tablet_manager.cc:1428] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260927 22:53:49.779634 16501 tablet_bootstrap.cc:492] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: No bootstrap required, opened a new log
I20260927 22:53:49.779767 16501 ts_tablet_manager.cc:1397] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:53:49.779785 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.772483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:40015)
0927 22:53:49.779247 (+ 6764us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.779461 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6641,"thread_start_us":6490,"threads_started":1}
I20260927 22:53:49.779744 16119 heartbeater.cc:503] Master 127.8.204.189:45951 was elected leader, sending a full tablet report...
I20260927 22:53:49.780550 16501 raft_consensus.cc:348] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:49.780817 16501 raft_consensus.cc:374] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:49.780910 16501 raft_consensus.cc:729] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f20ca8f443449d4b8f2075e3ad3b8ec, State: Initialized, Role: FOLLOWER
I20260927 22:53:49.781155 16501 consensus_queue.cc:260] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:49.781839 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.780836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:33633)
0927 22:53:49.781554 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.781688 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594}
I20260927 22:53:49.783211 16501 ts_tablet_manager.cc:1428] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:53:49.783290 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.782278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:38863)
0927 22:53:49.782795 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.782970 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":278,"threads_started":1}
I20260927 22:53:49.784366 16408 heartbeater.cc:503] Master 127.8.204.189:45951 was elected leader, sending a full tablet report...
I20260927 22:53:49.784727 16504 tablet_bootstrap.cc:492] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: No bootstrap required, opened a new log
I20260927 22:53:49.784929 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.783869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:58203)
0927 22:53:49.784443 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.784632 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":289,"threads_started":1}
I20260927 22:53:49.784919 16504 ts_tablet_manager.cc:1397] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 22:53:49.785840 16504 raft_consensus.cc:348] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:49.786088 16504 raft_consensus.cc:374] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:49.786199 16504 raft_consensus.cc:729] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0227056099ad4354bde4a3cae8940c96, State: Initialized, Role: FOLLOWER
I20260927 22:53:49.786474 16504 consensus_queue.cc:260] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:49.787213 16504 ts_tablet_manager.cc:1428] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:49.789029 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.787426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:47165)
0927 22:53:49.788519 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.788708 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":959,"thread_start_us":276,"threads_started":1}
I20260927 22:53:49.789207 16267 heartbeater.cc:503] Master 127.8.204.189:45951 was elected leader, sending a full tablet report...
I20260927 22:53:49.790509 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:49.789586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:58991)
0927 22:53:49.790093 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:49.790259 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":240,"threads_started":1}
W20260927 22:53:49.901769 16120 tablet.cc:2369] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:49.993228 16268 tablet.cc:2369] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:50.025399 16503 raft_consensus.cc:482] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:50.025625 16503 raft_consensus.cc:504] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:50.027108 16503 leader_election.cc:288] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231), 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645)
I20260927 22:53:50.033448 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.027644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:49343 (local address 127.8.204.130:42231)
0927 22:53:50.027848 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:50.027855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:50.027876 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:50.028138 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:50.028144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:50.028226 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:50.028342 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:50.028887 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.029745 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.031630 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.032450 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.032566 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.032623 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:50.033043 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:53:50.033176 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:53:50.033494 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.027710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:38977 (local address 127.8.204.131:44645)
0927 22:53:50.027885 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:50.027891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:50.028413 (+ 522us) server_negotiation.cc:407] Connection header received
0927 22:53:50.028600 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:50.028605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:50.028685 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:50.028783 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:50.029333 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.030088 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.031799 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.032598 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.032757 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.032836 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:50.033115 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 22:53:50.033246 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:53:50.033560 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.027463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:42231 (local address 127.8.204.129:49343)
0927 22:53:50.027717 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.027744 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:50.027929 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:50.028361 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:50.028387 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:50.028427 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:50.028745 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:50.028760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.029873 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.029879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:50.031485 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:50.031496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.032656 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.032666 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:50.032877 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.032922 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:53:50.032993 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 22:53:50.033162 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":49,"spinlock_wait_cycles":1792}
I20260927 22:53:50.033867 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.027547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:44645 (local address 127.8.204.129:38977)
0927 22:53:50.028302 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.028320 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:50.028447 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:50.028799 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:50.028802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:50.028839 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:50.029188 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:50.029199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.030213 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.030217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:50.031657 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:50.031670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.032789 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.032794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:50.032938 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.032958 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 22:53:50.033064 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:53:50.033254 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":54,"spinlock_wait_cycles":8960,"thread_start_us":480,"threads_started":1}
I20260927 22:53:50.034240 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aabef284bbd4510b0aecab0988153d1" candidate_uuid: "58fd2a85f6534816b8a77be4c332c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0227056099ad4354bde4a3cae8940c96" is_pre_election: true
I20260927 22:53:50.034541 16220 raft_consensus.cc:2454] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58fd2a85f6534816b8a77be4c332c34a in term 0.
I20260927 22:53:50.034729 16361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aabef284bbd4510b0aecab0988153d1" candidate_uuid: "58fd2a85f6534816b8a77be4c332c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" is_pre_election: true
I20260927 22:53:50.035081 16361 raft_consensus.cc:2454] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58fd2a85f6534816b8a77be4c332c34a in term 0.
I20260927 22:53:50.035066 16007 leader_election.cc:302] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0227056099ad4354bde4a3cae8940c96, 58fd2a85f6534816b8a77be4c332c34a; no voters:
I20260927 22:53:50.035454 16503 raft_consensus.cc:2781] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:50.035619 16503 raft_consensus.cc:482] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:50.035727 16503 raft_consensus.cc:3037] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:50.037809 16503 raft_consensus.cc:504] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:50.038357 16503 leader_election.cc:288] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [CANDIDATE]: Term 1 election: Requested vote from peers 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231), 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645)
I20260927 22:53:50.038834 16361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aabef284bbd4510b0aecab0988153d1" candidate_uuid: "58fd2a85f6534816b8a77be4c332c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec"
I20260927 22:53:50.038843 16220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aabef284bbd4510b0aecab0988153d1" candidate_uuid: "58fd2a85f6534816b8a77be4c332c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0227056099ad4354bde4a3cae8940c96"
I20260927 22:53:50.039016 16361 raft_consensus.cc:3037] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:50.039044 16220 raft_consensus.cc:3037] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:50.041071 16220 raft_consensus.cc:2454] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58fd2a85f6534816b8a77be4c332c34a in term 1.
I20260927 22:53:50.041090 16361 raft_consensus.cc:2454] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58fd2a85f6534816b8a77be4c332c34a in term 1.
I20260927 22:53:50.041492 16007 leader_election.cc:302] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0227056099ad4354bde4a3cae8940c96, 58fd2a85f6534816b8a77be4c332c34a; no voters:
I20260927 22:53:50.041787 16503 raft_consensus.cc:2781] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:50.042179 16503 raft_consensus.cc:686] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 1 LEADER]: Becoming Leader. State: Replica: 58fd2a85f6534816b8a77be4c332c34a, State: Running, Role: LEADER
I20260927 22:53:50.042688 16503 consensus_queue.cc:237] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } }
I20260927 22:53:50.043195 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.042770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:34981)
0927 22:53:50.042918 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.043048 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 22:53:50.046276 15752 catalog_manager.cc:5495] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129). New cstate: current_term: 1 leader_uuid: "58fd2a85f6534816b8a77be4c332c34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd2a85f6534816b8a77be4c332c34a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260927 22:53:50.130064 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.124990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.8.204.129:41189)
0927 22:53:50.125199 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:50.125204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:50.125225 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:50.125348 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:50.125351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:50.125423 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:50.125565 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:50.125914 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.126623 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.128025 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.128442 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.128575 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.128982 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:50.129494 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:50.129649 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:50.129744 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 22:53:50.129819 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:53:50.133921 16220 raft_consensus.cc:1261] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Refusing update from remote peer 58fd2a85f6534816b8a77be4c332c34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:50.134318 16361 raft_consensus.cc:1261] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Refusing update from remote peer 58fd2a85f6534816b8a77be4c332c34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:50.134681 16515 consensus_queue.cc:1035] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0227056099ad4354bde4a3cae8940c96" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:50.135053 16503 consensus_queue.cc:1035] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f20ca8f443449d4b8f2075e3ad3b8ec" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:50.137185 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.136664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:39059)
0927 22:53:50.136918 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.137010 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260927 22:53:50.137741 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.137310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:57275)
0927 22:53:50.137531 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.137632 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 22:53:50.143023 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.142385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.130:56161)
0927 22:53:50.142641 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.142799 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 22:53:50.146324 16519 mvcc.cc:205] Tried to move back new op lower bound from 7334091285022453760 to 7334091284658761728. Current Snapshot: MvccSnapshot[applied={T|T < 7334091285022453760}]
I20260927 22:53:50.146574 16520 mvcc.cc:205] Tried to move back new op lower bound from 7334091285022453760 to 7334091284658761728. Current Snapshot: MvccSnapshot[applied={T|T < 7334091285022453760 or (T in {7334091285022453760})}]
I20260927 22:53:50.152709 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.152238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:34321)
0927 22:53:50.152422 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.152533 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 22:53:50.158039 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.157190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.129:54891)
0927 22:53:50.157750 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.157864 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377}
I20260927 22:53:50.160852 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.160339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.8.204.131:42027)
0927 22:53:50.160574 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.160696 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 22:53:50.164021 15752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:50.164268 15752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:50.170651 15752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0c24fd1fec242dfaab8755016f0d8da: Sending DeleteTablet for 3 replicas of tablet 9aabef284bbd4510b0aecab0988153d1
I20260927 22:53:50.171645 16052 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aabef284bbd4510b0aecab0988153d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:50 UTC) from {username='oryx', principal='oryx/127.8.204.189@KRBTEST.COM'} at 127.0.0.1:46692
I20260927 22:53:50.171761 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aabef284bbd4510b0aecab0988153d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:50 UTC) from {username='oryx', principal='oryx/127.8.204.189@KRBTEST.COM'} at 127.0.0.1:40518
I20260927 22:53:50.172068 16341 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aabef284bbd4510b0aecab0988153d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:50 UTC) from {username='oryx', principal='oryx/127.8.204.189@KRBTEST.COM'} at 127.0.0.1:60504
I20260927 22:53:50.173784 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15912
I20260927 22:53:50.180562 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.179977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41434)
0927 22:53:50.180231 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.180353 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 22:53:50.181967 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.181453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44551 (local address 127.0.0.1:41436)
0927 22:53:50.181679 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:50.181791 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:44551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 22:53:50.182507 16530 tablet_replica.cc:330] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: stopping tablet replica
I20260927 22:53:50.182700 16530 raft_consensus.cc:2229] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:50.183091 16530 raft_consensus.cc:2258] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:50.184175 16530 ts_tablet_manager.cc:1905] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:50.184845 16531 tablet_replica.cc:330] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: stopping tablet replica
I20260927 22:53:50.185175 16531 raft_consensus.cc:2229] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:50.185570 16531 raft_consensus.cc:2258] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:50.186820 16531 ts_tablet_manager.cc:1905] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:50.190629 16530 ts_tablet_manager.cc:1918] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:50.190752 16530 log.cc:1196] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/9aabef284bbd4510b0aecab0988153d1
I20260927 22:53:50.191206 16530 ts_tablet_manager.cc:1939] T 9aabef284bbd4510b0aecab0988153d1 P 0227056099ad4354bde4a3cae8940c96: Deleting consensus metadata
I20260927 22:53:50.191397 16529 tablet_replica.cc:330] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: stopping tablet replica
I20260927 22:53:50.192253 15738 catalog_manager.cc:4841] TS 0227056099ad4354bde4a3cae8940c96 (127.8.204.130:42231): tablet 9aabef284bbd4510b0aecab0988153d1 (table test-table [id=4c8ac03d6aa547dd8810df7d05c2c807]) successfully deleted
I20260927 22:53:50.194221 16531 ts_tablet_manager.cc:1918] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:50.194352 16531 log.cc:1196] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/9aabef284bbd4510b0aecab0988153d1
I20260927 22:53:50.194751 16531 ts_tablet_manager.cc:1939] T 9aabef284bbd4510b0aecab0988153d1 P 2f20ca8f443449d4b8f2075e3ad3b8ec: Deleting consensus metadata
I20260927 22:53:50.191641 16529 raft_consensus.cc:2229] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:50.195711 15736 catalog_manager.cc:4841] TS 2f20ca8f443449d4b8f2075e3ad3b8ec (127.8.204.131:44645): tablet 9aabef284bbd4510b0aecab0988153d1 (table test-table [id=4c8ac03d6aa547dd8810df7d05c2c807]) successfully deleted
I20260927 22:53:50.195960 16529 raft_consensus.cc:2258] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:50.197013 16529 ts_tablet_manager.cc:1905] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:50.203958 16529 ts_tablet_manager.cc:1918] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:50.204093 16529 log.cc:1196] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/9aabef284bbd4510b0aecab0988153d1
I20260927 22:53:50.204564 16529 ts_tablet_manager.cc:1939] T 9aabef284bbd4510b0aecab0988153d1 P 58fd2a85f6534816b8a77be4c332c34a: Deleting consensus metadata
I20260927 22:53:50.205633 15736 catalog_manager.cc:4841] TS 58fd2a85f6534816b8a77be4c332c34a (127.8.204.129:41189): tablet 9aabef284bbd4510b0aecab0988153d1 (table test-table [id=4c8ac03d6aa547dd8810df7d05c2c807]) successfully deleted
I20260927 22:53:50.208420 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.175561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.8.204.129:41189)
0927 22:53:50.175786 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:50.175795 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:50.191750 (+ 15955us) server_negotiation.cc:407] Connection header received
0927 22:53:50.191912 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:50.191917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:50.195438 (+ 3521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:50.195559 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:50.197243 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.197972 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.199170 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.199729 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.199864 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.202303 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:50.202313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:50.202331 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:50.202376 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:50.205656 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:50.206354 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:50.206363 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:50.206366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:50.206541 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:50.206913 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:50.206923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:50.206927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:50.207311 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:50.207613 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:50.207881 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 22:53:50.208080 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:53:50.209085 16112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:50.324542 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15912
I20260927 22:53:50.342685 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 16132
I20260927 22:53:50.354959 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.343404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.8.204.130:42231)
0927 22:53:50.343564 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:50.343568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:50.343693 (+ 125us) server_negotiation.cc:407] Connection header received
0927 22:53:50.343830 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:50.343834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:50.343891 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:50.343983 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:50.345193 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.345903 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.347062 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.347577 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.347675 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.350087 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:50.350098 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:50.350112 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:50.350138 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:50.352750 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:50.353326 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:50.353332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:50.353334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:50.353442 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:50.353752 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:50.353755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:50.353757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:50.354058 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:50.354273 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:50.354553 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 22:53:50.354705 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:53:50.355643 16260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:50.448194 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 16132
I20260927 22:53:50.466922 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 16278
I20260927 22:53:50.480458 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:50.467677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.8.204.131:44645)
0927 22:53:50.467856 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:50.467862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:50.468022 (+ 160us) server_negotiation.cc:407] Connection header received
0927 22:53:50.468243 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:50.468247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:50.468310 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:50.468412 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:50.469698 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.470381 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.471736 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:50.472103 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:50.472236 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:50.474796 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:50.474809 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:50.474824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:50.474861 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:50.477729 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:50.478385 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:50.478392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:50.478394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:50.478523 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:50.478900 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:50.478905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:50.478908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:50.479252 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:50.479655 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:50.479921 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:53:50.480135 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 22:53:50.481097 16401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:50.598094 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 16278
I20260927 22:53:50.620895 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15616
I20260927 22:53:50.621937 15782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:50.755940 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15616
I20260927 22:53:50.779112 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 15716
I20260927 22:53:50.780133 15970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:50.894757 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 15716
2026-09-27T22:53:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9520 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:50 dist-test-slave-40vx krb5kdc[16541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:50 dist-test-slave-40vx krb5kdc[16541](info): set up 2 sockets
Sep 27 22:53:50 dist-test-slave-40vx krb5kdc[16541](info): commencing operation
krb5kdc: starting...
W20260927 22:53:51.976188 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:52 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab.
Entry for principal oryx/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab.
I20260927 22:53:52.182448 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44219
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab
--principal=oryx/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:52.498420 16554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:52.498780 16554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:52.498931 16554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:52.506104 16554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:52.506242 16554 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:53:52.506332 16554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:52.506448 16554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:52.506549 16554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:52.506637 16554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:52.516284 16554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44219
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.190.keytab
--principal=oryx/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:52.517623 16554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:52.518932 16554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:52.530380 16559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:52.530444 16560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:52.530467 16562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:52.531468 16554 server_base.cc:1034] running on GCE node
I20260927 22:53:52.532177 16554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:52.533558 16554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:52.534775 16554 hybrid_clock.cc:648] HybridClock initialized: now 1790549632534708 us; error 38 us; skew 500 ppm
I20260927 22:53:52.535104 16554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:52 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549632, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:52.538223 16554 init.cc:377] Logged in from keytab as oryx/127.8.204.190@KRBTEST.COM (short username oryx)
I20260927 22:53:52.539644 16554 webserver.cc:460] Webserver started at http://127.8.204.190:33373/ using document root <none> and password file <none>
I20260927 22:53:52.540324 16554 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:52.540431 16554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:52.540755 16554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:52.543648 16554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "2a831b2e61e142648ee6e8abed13a6d5"
format_stamp: "Formatted at 2026-09-27 22:53:52 on dist-test-slave-40vx"
server_key: "5ae7e06c091fd7ab200dcba8703c7394"
server_key_iv: "2fda899bf52e683ba4012e5cbb968453"
server_key_version: "encryptionkey@0"
I20260927 22:53:52.544411 16554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "2a831b2e61e142648ee6e8abed13a6d5"
format_stamp: "Formatted at 2026-09-27 22:53:52 on dist-test-slave-40vx"
server_key: "5ae7e06c091fd7ab200dcba8703c7394"
server_key_iv: "2fda899bf52e683ba4012e5cbb968453"
server_key_version: "encryptionkey@0"
I20260927 22:53:52.549278 16554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:53:52.552565 16569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:52.553829 16554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:53:52.554005 16554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "2a831b2e61e142648ee6e8abed13a6d5"
format_stamp: "Formatted at 2026-09-27 22:53:52 on dist-test-slave-40vx"
server_key: "5ae7e06c091fd7ab200dcba8703c7394"
server_key_iv: "2fda899bf52e683ba4012e5cbb968453"
server_key_version: "encryptionkey@0"
I20260927 22:53:52.554260 16554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:52.600522 16554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:52.604677 16554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:52.604960 16554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:52.626021 16554 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:44219
I20260927 22:53:52.626029 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:44219 every 8 connection(s)
I20260927 22:53:52.627477 16554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:53:52.630334 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 16554
I20260927 22:53:52.630530 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:53:52.631335 9010 external_mini_cluster.cc:1428] Setting key 70cdca462335fd810a27e1825a1659be
I20260927 22:53:52.631567 16631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:52.645742 16631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Bootstrap starting.
Sep 27 22:53:52 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:52.649549 16631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:52.650867 16631 log.cc:824] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:52.654172 16631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: No bootstrap required, opened a new log
I20260927 22:53:52.655573 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:52.633432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.8.204.190:44219)
0927 22:53:52.634248 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:52.634262 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:52.634301 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:53:52.636212 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:52.636252 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:52.636638 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:52.636967 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:52.638650 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:52.640032 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:52.641744 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:52.642427 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:52.642578 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:52.647126 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:52.647138 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:52.647159 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:52.647224 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:52.651097 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:52.652000 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:52.652006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:52.652011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:52.652200 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:52.652611 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:52.652623 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:52.652625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:52.653230 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:52.653502 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:52.653877 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 22:53:52.654275 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":361,"threads_started":1}
I20260927 22:53:52.662629 16631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a831b2e61e142648ee6e8abed13a6d5" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44219 } }
I20260927 22:53:52.663000 16631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:52.663077 16631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a831b2e61e142648ee6e8abed13a6d5, State: Initialized, Role: FOLLOWER
I20260927 22:53:52.663611 16631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a831b2e61e142648ee6e8abed13a6d5" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44219 } }
I20260927 22:53:52.663815 16631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:53:52.663908 16631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:53:52.664027 16631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:52.666033 16631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a831b2e61e142648ee6e8abed13a6d5" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44219 } }
I20260927 22:53:52.666504 16631 leader_election.cc:302] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a831b2e61e142648ee6e8abed13a6d5; no voters:
I20260927 22:53:52.667001 16631 leader_election.cc:288] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:53:52.667171 16636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:52.667928 16636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [term 1 LEADER]: Becoming Leader. State: Replica: 2a831b2e61e142648ee6e8abed13a6d5, State: Running, Role: LEADER
I20260927 22:53:52.668484 16636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a831b2e61e142648ee6e8abed13a6d5" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44219 } }
I20260927 22:53:52.669121 16631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:53:52.671341 16637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a831b2e61e142648ee6e8abed13a6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a831b2e61e142648ee6e8abed13a6d5" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44219 } } }
I20260927 22:53:52.671717 16637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:52.671443 16638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a831b2e61e142648ee6e8abed13a6d5. Latest consensus state: current_term: 1 leader_uuid: "2a831b2e61e142648ee6e8abed13a6d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a831b2e61e142648ee6e8abed13a6d5" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44219 } } }
I20260927 22:53:52.672039 16638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:53:52.675767 16642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:53:52.680850 16642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:53:52.689343 16642 catalog_manager.cc:1347] Generated new cluster ID: 81db4748c0ba43d08e9c46e2d6d10808
I20260927 22:53:52.689445 16642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:53:52.698818 16642 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:53:52.700080 16642 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:53:52.705060 16642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Generated new TSK 0
I20260927 22:53:52.705780 16642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab.
Entry for principal oryx/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab.
I20260927 22:53:52.766602 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44219
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:53.097281 16657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:53.097525 16657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:53.097692 16657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:53.104436 16657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:53.104558 16657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:53.104740 16657 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:53.104871 16657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:53:53.114207 16657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:53.115449 16657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:53.116707 16657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:53.129256 16665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:53.129366 16662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:53.129326 16663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:53.129930 16657 server_base.cc:1034] running on GCE node
I20260927 22:53:53.130421 16657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:53.131204 16657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:53.132413 16657 hybrid_clock.cc:648] HybridClock initialized: now 1790549633132337 us; error 49 us; skew 500 ppm
I20260927 22:53:53.132709 16657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:53 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549633, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:53.135905 16657 init.cc:377] Logged in from keytab as oryx/127.8.204.129@KRBTEST.COM (short username oryx)
I20260927 22:53:53.137336 16657 webserver.cc:460] Webserver started at http://127.8.204.129:44689/ using document root <none> and password file <none>
I20260927 22:53:53.138032 16657 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:53.138160 16657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:53.138485 16657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:53.141340 16657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "c83995cf1fea46f98178b068465a2b2a"
format_stamp: "Formatted at 2026-09-27 22:53:53 on dist-test-slave-40vx"
server_key: "fe095e0c89e2248e666c66d86e1eb9ba"
server_key_iv: "45d3c28577e40a2eb07b13f6b2c0dfc8"
server_key_version: "encryptionkey@0"
I20260927 22:53:53.142124 16657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "c83995cf1fea46f98178b068465a2b2a"
format_stamp: "Formatted at 2026-09-27 22:53:53 on dist-test-slave-40vx"
server_key: "fe095e0c89e2248e666c66d86e1eb9ba"
server_key_iv: "45d3c28577e40a2eb07b13f6b2c0dfc8"
server_key_version: "encryptionkey@0"
I20260927 22:53:53.147113 16657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 22:53:53.150538 16672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:53.151893 16657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:53:53.152076 16657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "c83995cf1fea46f98178b068465a2b2a"
format_stamp: "Formatted at 2026-09-27 22:53:53 on dist-test-slave-40vx"
server_key: "fe095e0c89e2248e666c66d86e1eb9ba"
server_key_iv: "45d3c28577e40a2eb07b13f6b2c0dfc8"
server_key_version: "encryptionkey@0"
I20260927 22:53:53.152300 16657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:53.181519 16657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:53.189770 16657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:53.189994 16657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:53.192528 16657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:53.192606 16657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:53.192775 16657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:53.192852 16657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:53 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549633, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:53.214783 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.193067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:34045 (local address 127.8.204.190:44219)
0927 22:53:53.193741 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:53.193747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:53.194783 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 22:53:53.196865 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:53.196872 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:53.196969 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:53.197095 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:53.199295 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.200565 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.203126 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.203835 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.203976 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.208272 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:53.208279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:53.208289 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:53.208320 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:53.211565 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.212539 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.212545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.212551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.212679 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.213105 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.213110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.213114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.213385 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:53.213534 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:53.214211 (+ 677us) server_negotiation.cc:299] Negotiation successful
0927 22:53:53.214428 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":444,"threads_started":1}
I20260927 22:53:53.216216 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.193778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.8.204.129:34045)
0927 22:53:53.194618 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:53.194665 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:53.196588 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:53.197358 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:53.197373 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:53.198183 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:53.199015 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.199067 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.200781 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.200792 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.202971 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.202983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.204085 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.204094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.204406 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.205211 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:53.205233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:53.207993 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:53.211696 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.211705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.211721 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.212402 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.212793 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.212799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.212803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.212989 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.213535 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:53.213544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:53.213869 (+ 325us) client_negotiation.cc:769] Sending connection context
0927 22:53:53.214135 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 22:53:53.214661 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":370,"threads_started":1}
I20260927 22:53:53.239585 16657 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:42915
I20260927 22:53:53.239645 16792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:42915 every 8 connection(s)
I20260927 22:53:53.240737 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.224804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:41859 (local address 127.8.204.190:44219)
0927 22:53:53.225059 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:53.225069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:53.225095 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:53:53.225256 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:53.225262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:53.225348 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:53.225470 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:53.226417 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.227731 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.230430 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.231116 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.231215 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.234017 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:53.234026 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:53.234032 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:53.234072 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:53.237865 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.238714 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.238724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.238730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.238895 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.239432 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.239440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.239446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.239797 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:53.239940 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:53.240357 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 22:53:53.240506 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:53:53.241070 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.224390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.8.204.129:41859)
0927 22:53:53.224946 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:53.224971 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:53.225101 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:53.225496 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:53.225502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:53.226001 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:53.226275 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.226283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.227929 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.227938 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.230261 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.230288 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.231343 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.231351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.231545 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.232206 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:53.232232 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:53.233734 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:53.237990 (+ 4256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.237995 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.238000 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.238556 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.239091 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.239094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.239098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.239265 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.239915 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:53.239920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:53.240111 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 22:53:53.240276 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:53:53.240610 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":464,"threads_started":1}
I20260927 22:53:53.241341 16657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:53:53.243312 16793 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44219
I20260927 22:53:53.243870 16793 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:53.244663 16793 heartbeater.cc:511] Master 127.8.204.190:44219 requested a full tablet report, sending...
I20260927 22:53:53.246419 16586 ts_manager.cc:194] Registered new tserver with Master: c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915)
I20260927 22:53:53.247885 16586 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:34045
I20260927 22:53:53.249218 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 16657
I20260927 22:53:53.249404 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:53:53.250008 9010 external_mini_cluster.cc:1428] Setting key d4237426a3c80ea44c464cf244349390
WARNING: no policy specified for oryx/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab.
Entry for principal oryx/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab.
I20260927 22:53:53.294631 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44219
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:53.606946 16797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:53.607201 16797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:53.607409 16797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:53.614538 16797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:53.614637 16797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:53.614742 16797 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:53.614874 16797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:53:53.624408 16797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:53.625717 16797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:53.626995 16797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:53.639521 16803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:53.639969 16802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:53.640445 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:53.640746 16797 server_base.cc:1034] running on GCE node
I20260927 22:53:53.641367 16797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:53.642270 16797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:53.643497 16797 hybrid_clock.cc:648] HybridClock initialized: now 1790549633643436 us; error 44 us; skew 500 ppm
I20260927 22:53:53.643787 16797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:53 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549633, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:53.646842 16797 init.cc:377] Logged in from keytab as oryx/127.8.204.130@KRBTEST.COM (short username oryx)
I20260927 22:53:53.648252 16797 webserver.cc:460] Webserver started at http://127.8.204.130:33969/ using document root <none> and password file <none>
I20260927 22:53:53.648910 16797 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:53.649035 16797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:53.649343 16797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:53.652272 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
format_stamp: "Formatted at 2026-09-27 22:53:53 on dist-test-slave-40vx"
server_key: "9ed3cbef92d2c406fb96d9e173601eb0"
server_key_iv: "3985b1a2c8fc40f06eafdaa3d4ef5005"
server_key_version: "encryptionkey@0"
I20260927 22:53:53.653020 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
format_stamp: "Formatted at 2026-09-27 22:53:53 on dist-test-slave-40vx"
server_key: "9ed3cbef92d2c406fb96d9e173601eb0"
server_key_iv: "3985b1a2c8fc40f06eafdaa3d4ef5005"
server_key_version: "encryptionkey@0"
I20260927 22:53:53.657867 16797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:53:53.661135 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:53.662343 16797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:53.662520 16797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
format_stamp: "Formatted at 2026-09-27 22:53:53 on dist-test-slave-40vx"
server_key: "9ed3cbef92d2c406fb96d9e173601eb0"
server_key_iv: "3985b1a2c8fc40f06eafdaa3d4ef5005"
server_key_version: "encryptionkey@0"
I20260927 22:53:53.662731 16797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:53.675688 16797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:53.679812 16797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:53.680084 16797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:53.682636 16797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:53.682731 16797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:53.682858 16797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:53.682935 16797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:53 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549633, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:53.705155 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.683234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:48747 (local address 127.8.204.190:44219)
0927 22:53:53.683470 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:53.683479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:53.684849 (+ 1370us) server_negotiation.cc:407] Connection header received
0927 22:53:53.687026 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:53.687032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:53.687119 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:53.687227 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:53.688676 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.689728 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.692314 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.692966 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.693151 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.698743 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:53.698748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:53.698751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:53.698783 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:53.701914 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.702834 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.702839 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.702842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.702964 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.703497 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.703501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.703503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.703775 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:53.703928 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:53.704598 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 22:53:53.704812 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:53:53.706254 16821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.683872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.8.204.130:48747)
0927 22:53:53.684686 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:53.684722 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:53.686744 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:53.687487 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:53.687495 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:53.687961 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:53.688496 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.688512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.689892 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.689897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.692149 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.692170 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.693260 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.693271 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.693648 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.694499 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:53.694536 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:53.698465 (+ 3929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:53.702053 (+ 3588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.702062 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.702080 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.702703 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.703083 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.703088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.703092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.703348 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.703907 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:53.703915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:53.704249 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 22:53:53.704501 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:53:53.704993 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":363,"threads_started":1}
I20260927 22:53:53.728221 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.712938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:34429 (local address 127.8.204.190:44219)
0927 22:53:53.713105 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:53.713108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:53.713766 (+ 658us) server_negotiation.cc:407] Connection header received
0927 22:53:53.713951 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:53.713961 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:53.714017 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:53.714114 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:53.714972 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.715774 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.718670 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.719409 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.719554 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.722089 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:53.722094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:53.722097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:53.722128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:53.725164 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.726165 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.726173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.726178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.726363 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:53.726747 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:53.726754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:53.726758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:53.727056 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:53.727218 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:53.727672 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:53:53.727883 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 22:53:53.728345 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:53.713099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.8.204.130:34429)
0927 22:53:53.713667 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:53.713682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:53.713831 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:53.714143 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:53.714148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:53.714567 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:53.714838 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.714846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.715977 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.715985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.718537 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:53.718555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:53.719633 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:53.719643 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:53.719919 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:53.720599 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:53.720630 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:53.721900 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:53.725311 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.725316 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.725323 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.726024 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.726508 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:53.726513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:53.726517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:53.726645 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:53.727185 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:53.727191 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:53.727417 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 22:53:53.727586 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 22:53:53.727900 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":328,"threads_started":1}
I20260927 22:53:53.741647 16797 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:43629
I20260927 22:53:53.741701 16931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:43629 every 8 connection(s)
I20260927 22:53:53.742869 16797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:53:53.744571 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 16797
I20260927 22:53:53.744735 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:53:53.745288 9010 external_mini_cluster.cc:1428] Setting key b4f9e1c5b8f8ee2cd1bcf3cb594a349a
I20260927 22:53:53.745398 16932 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44219
I20260927 22:53:53.745615 16932 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:53.746407 16932 heartbeater.cc:511] Master 127.8.204.190:44219 requested a full tablet report, sending...
I20260927 22:53:53.747829 16586 ts_manager.cc:194] Registered new tserver with Master: e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:53.749001 16586 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:48747
WARNING: no policy specified for oryx/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab.
Entry for principal oryx/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab.
I20260927 22:53:53.792210 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44219
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:54.114405 16936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:54.114729 16936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:54.114893 16936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:54.122304 16936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:54.122452 16936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:54.122589 16936 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:53:54.122720 16936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:53:54.132579 16936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.204.190:44219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:54.133862 16936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:54.135143 16936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:54.148270 16941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:54.148308 16942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:54.148518 16944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:54.149247 16936 server_base.cc:1034] running on GCE node
I20260927 22:53:54.149825 16936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:54.150694 16936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:54.151922 16936 hybrid_clock.cc:648] HybridClock initialized: now 1790549634151842 us; error 60 us; skew 500 ppm
I20260927 22:53:54.152212 16936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:53:54 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549634, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:53:54.155184 16936 init.cc:377] Logged in from keytab as oryx/127.8.204.131@KRBTEST.COM (short username oryx)
I20260927 22:53:54.156775 16936 webserver.cc:460] Webserver started at http://127.8.204.131:38965/ using document root <none> and password file <none>
I20260927 22:53:54.157503 16936 fs_manager.cc:359] Metadata directory not provided
I20260927 22:53:54.157634 16936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:53:54.157970 16936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:53:54.160930 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "eb5f825266d64599a3147bc88067cbb3"
format_stamp: "Formatted at 2026-09-27 22:53:54 on dist-test-slave-40vx"
server_key: "14db7e1cfac1807b7551fe9a4bb812f2"
server_key_iv: "c737f18b458e6ac8101c116c192a40cb"
server_key_version: "encryptionkey@0"
I20260927 22:53:54.161679 16936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "eb5f825266d64599a3147bc88067cbb3"
format_stamp: "Formatted at 2026-09-27 22:53:54 on dist-test-slave-40vx"
server_key: "14db7e1cfac1807b7551fe9a4bb812f2"
server_key_iv: "c737f18b458e6ac8101c116c192a40cb"
server_key_version: "encryptionkey@0"
I20260927 22:53:54.167315 16936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:53:54.170966 16951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:54.172511 16936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:53:54.172804 16936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "eb5f825266d64599a3147bc88067cbb3"
format_stamp: "Formatted at 2026-09-27 22:53:54 on dist-test-slave-40vx"
server_key: "14db7e1cfac1807b7551fe9a4bb812f2"
server_key_iv: "c737f18b458e6ac8101c116c192a40cb"
server_key_version: "encryptionkey@0"
I20260927 22:53:54.173054 16936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:53:54.186764 16936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:53:54.191388 16936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:53:54.191704 16936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:53:54.194444 16936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:53:54.194671 16936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:53:54.194864 16936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:53:54.194939 16936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:53:54 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549634, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:54.217497 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:54.194754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:42479 (local address 127.8.204.190:44219)
0927 22:53:54.194962 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:54.194969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:54.196638 (+ 1669us) server_negotiation.cc:407] Connection header received
0927 22:53:54.199602 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:54.199609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:54.199699 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:54.199808 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:54.202162 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.203235 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.205543 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.206157 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.206282 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:54.210469 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:54.210475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:54.210478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:54.210508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:54.213476 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:54.214409 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:54.214417 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:54.214424 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:54.214634 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:54.215097 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:54.215105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:54.215111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:54.215513 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:54.215682 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:54.216684 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0927 22:53:54.216981 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:53:54.218082 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:54.195470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.8.204.131:42479)
0927 22:53:54.196428 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:54.196495 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:54.199224 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:54.200248 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:54.200261 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:54.201034 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:54.201910 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:54.201944 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.203419 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.203427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:54.205400 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:54.205411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.206406 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.206417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:54.206614 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:54.207434 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:54.207455 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:54.210240 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:54.213606 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:54.213616 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:54.213634 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:54.214278 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:54.214770 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:54.214776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:54.214780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:54.214977 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:54.215658 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:54.215666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:54.216156 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 22:53:54.216503 (+ 347us) client_negotiation.cc:240] Negotiation successful
0927 22:53:54.216917 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":331,"threads_started":1}
I20260927 22:53:54.238842 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:54.223817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:41969 (local address 127.8.204.190:44219)
0927 22:53:54.224016 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:54.224025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:54.224738 (+ 713us) server_negotiation.cc:407] Connection header received
0927 22:53:54.224977 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:54.224983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:54.225079 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:54.225189 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:54.227034 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.228035 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.229952 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.230383 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.230505 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:54.232800 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:54.232806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:54.232811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:54.232846 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:54.236209 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:54.236857 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:54.236870 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:54.236872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:54.236989 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:54.237427 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:54.237431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:54.237433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:54.237687 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:54.237833 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:54.238237 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:53:54.238455 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:53:54.238931 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:54.223887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.8.204.131:41969)
0927 22:53:54.224631 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:54.224651 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:54.224821 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:54.225408 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:54.225416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:54.226234 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:54.226840 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:54.226857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.228173 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.228179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:54.229821 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:54.229832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.230565 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.230572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:54.230738 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:54.231400 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:54.231417 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:54.232534 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:54.236341 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:54.236347 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:54.236351 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:54.236736 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:54.237096 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:54.237101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:54.237105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:54.237291 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:54.237811 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:54.237816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:54.237988 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 22:53:54.238129 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:53:54.238464 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":439,"threads_started":1}
I20260927 22:53:54.250635 16793 heartbeater.cc:503] Master 127.8.204.190:44219 was elected leader, sending a full tablet report...
I20260927 22:53:54.251861 16936 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36329
I20260927 22:53:54.251914 17070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36329 every 8 connection(s)
I20260927 22:53:54.253201 16936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:53:54.255069 17071 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44219
I20260927 22:53:54.255321 17071 heartbeater.cc:464] Registering TS with master...
I20260927 22:53:54.256059 17071 heartbeater.cc:511] Master 127.8.204.190:44219 requested a full tablet report, sending...
I20260927 22:53:54.257408 16586 ts_manager.cc:194] Registered new tserver with Master: eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329)
I20260927 22:53:54.258205 16586 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:42479
I20260927 22:53:54.261004 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 16936
I20260927 22:53:54.261160 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:53:54.261582 9010 external_mini_cluster.cc:1428] Setting key 3ef15436d0ebaa515f7bd4b0619238d8
I20260927 22:53:54.264564 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:53:54 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549634, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:53:54 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:53:54 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:53:54.752332 16932 heartbeater.cc:503] Master 127.8.204.190:44219 was elected leader, sending a full tablet report...
I20260927 22:53:55.260627 17071 heartbeater.cc:503] Master 127.8.204.190:44219 was elected leader, sending a full tablet report...
W20260927 22:53:55.363230 17079 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.057s user 0.001s sys 0.014s
W20260927 22:53:55.364337 16681 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:54.300818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.8.204.190:44219)
0927 22:53:54.301045 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:54.301053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:54.301274 (+ 221us) server_negotiation.cc:407] Connection header received
0927 22:53:54.301496 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:54.301500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:54.301563 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:54.301682 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:54.302946 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.303755 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.305013 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:54.305441 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:54.305575 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.364002 (+1058427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48786: BlockingRecv error: recv got EOF from 127.0.0.1:48786 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:53:55 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549634, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:55.382174 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.368598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.8.204.190:44219)
0927 22:53:55.368824 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.368831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.369118 (+ 287us) server_negotiation.cc:407] Connection header received
0927 22:53:55.369261 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.369264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.369352 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.369491 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:55.370760 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.371542 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.372736 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.373122 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.373219 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.377037 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:55.377042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:55.377045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:55.377074 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:55.379713 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:55.380333 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:55.380338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:55.380342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:55.380460 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:55.380828 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:55.380833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:55.380837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:55.381097 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:55.381388 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.381646 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.381861 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:53:55.385893 16586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:55.392693 16586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:55.419094 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.405315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.8.204.129:42915)
0927 22:53:55.406094 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.406106 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.407060 (+ 954us) server_negotiation.cc:407] Connection header received
0927 22:53:55.407536 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.407597 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.407907 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.408123 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.409470 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.411520 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.416093 (+ 4573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.417606 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.417773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.417962 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.418592 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.418781 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":456,"threads_started":1}
I20260927 22:53:55.419160 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.404878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42915 (local address 127.0.0.1:34336)
0927 22:53:55.406877 (+ 1999us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.406928 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.407239 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.408529 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.408540 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.408584 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.409177 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.409194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.411727 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.411736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.414906 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.414925 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.418075 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.418081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.418295 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.418366 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.418532 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.418740 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1763,"thread_start_us":1499,"threads_started":1}
I20260927 22:53:55.423458 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.404349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:43629 (local address 127.0.0.1:50054)
0927 22:53:55.405268 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.405311 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.405495 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.416128 (+ 10633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.416132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.416160 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.416487 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.416497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.418391 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.418397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.420727 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.420742 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.422630 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.422637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.422799 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.422849 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.422911 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.423035 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":1,"thread_start_us":446,"threads_started":1}
I20260927 22:53:55.423651 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.406262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.8.204.131:36329)
0927 22:53:55.416160 (+ 9898us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.416166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.416187 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:53:55.416256 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.416261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.416401 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.416591 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.417088 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.418512 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.421008 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.422393 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.422529 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.422744 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.423199 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.423389 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9749,"thread_start_us":9624,"threads_started":1}
I20260927 22:53:55.423719 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.405693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36329 (local address 127.0.0.1:44496)
0927 22:53:55.415784 (+ 10091us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.415801 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.415905 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.416574 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.416578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.416603 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.416916 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.416925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.418687 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.418692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.420853 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.420863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.422869 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.422875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.423027 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.423052 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.423133 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.423255 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9992,"thread_start_us":9870,"threads_started":1}
I20260927 22:53:55.422989 16723 tablet_service.cc:1467] Processing CreateTablet for tablet 882f22285f27435cb79e539126c5253a (DEFAULT_TABLE table=test-table [id=4d393c0f2f974fec9f45b7411824f451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:55.424409 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.404523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.8.204.130:43629)
0927 22:53:55.415487 (+ 10964us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.415499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.415531 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:53:55.415638 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.415648 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.415906 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.416669 (+ 763us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.416854 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.418271 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.420928 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.422391 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.423680 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.423895 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.423969 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.424162 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10807,"thread_start_us":10658,"threads_started":1}
I20260927 22:53:55.424661 16723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882f22285f27435cb79e539126c5253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.427109 16862 tablet_service.cc:1467] Processing CreateTablet for tablet 882f22285f27435cb79e539126c5253a (DEFAULT_TABLE table=test-table [id=4d393c0f2f974fec9f45b7411824f451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:55.427752 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 882f22285f27435cb79e539126c5253a (DEFAULT_TABLE table=test-table [id=4d393c0f2f974fec9f45b7411824f451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:55.428381 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882f22285f27435cb79e539126c5253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.428941 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 882f22285f27435cb79e539126c5253a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.438242 17096 tablet_bootstrap.cc:492] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Bootstrap starting.
I20260927 22:53:55.440140 17097 tablet_bootstrap.cc:492] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Bootstrap starting.
I20260927 22:53:55.442240 17096 tablet_bootstrap.cc:654] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.443158 17098 tablet_bootstrap.cc:492] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Bootstrap starting.
I20260927 22:53:55.443552 17097 tablet_bootstrap.cc:654] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.444051 17096 log.cc:824] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:55.444823 17097 log.cc:824] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:55.446782 17098 tablet_bootstrap.cc:654] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.447477 17096 tablet_bootstrap.cc:492] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: No bootstrap required, opened a new log
I20260927 22:53:55.447775 17096 ts_tablet_manager.cc:1397] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:53:55.448108 17098 log.cc:824] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:53:55.449494 17097 tablet_bootstrap.cc:492] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: No bootstrap required, opened a new log
I20260927 22:53:55.449761 17097 ts_tablet_manager.cc:1397] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260927 22:53:55.450626 17098 tablet_bootstrap.cc:492] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: No bootstrap required, opened a new log
I20260927 22:53:55.450865 17098 ts_tablet_manager.cc:1397] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:53:55.458323 17096 raft_consensus.cc:348] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.458755 17096 raft_consensus.cc:374] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.458880 17096 raft_consensus.cc:729] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67022eedb6f407ca25ff12dd73ae2e8, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.458671 17098 raft_consensus.cc:348] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.459113 17098 raft_consensus.cc:374] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.459286 17098 raft_consensus.cc:729] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.459514 17096 consensus_queue.cc:260] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.459905 17098 consensus_queue.cc:260] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.459918 17097 raft_consensus.cc:348] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.460402 17097 raft_consensus.cc:374] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.460528 17097 raft_consensus.cc:729] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83995cf1fea46f98178b068465a2b2a, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.461264 17096 ts_tablet_manager.cc:1428] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:53:55.461184 17097 consensus_queue.cc:260] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.462782 17098 ts_tablet_manager.cc:1428] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 22:53:55.465126 17097 ts_tablet_manager.cc:1428] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260927 22:53:55.495193 16794 tablet.cc:2369] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:53:55.495857 16933 tablet.cc:2369] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:55.497349 17104 raft_consensus.cc:482] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:55.497609 17104 raft_consensus.cc:504] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.499104 17104 leader_election.cc:288] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629), eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329)
W20260927 22:53:55.505343 17072 tablet.cc:2369] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:53:55.506875 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.499542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:35951 (local address 127.8.204.130:43629)
0927 22:53:55.499705 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.499710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.500320 (+ 610us) server_negotiation.cc:407] Connection header received
0927 22:53:55.500549 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.500554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.500657 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.500709 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.501351 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.502064 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.504824 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.506013 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.506174 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.506246 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.506490 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.506638 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:53:55.507042 17105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.499573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:43629 (local address 127.8.204.129:35951)
0927 22:53:55.500166 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.500209 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.500365 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.500861 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.500864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.500882 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.501221 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.501229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.502200 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.502208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.504659 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.504673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.506194 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.506198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.506300 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.506346 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.506451 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.506606 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":100,"spinlock_wait_cycles":55680,"thread_start_us":300,"threads_started":1}
I20260927 22:53:55.507175 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.500539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:41191 (local address 127.8.204.131:36329)
0927 22:53:55.500731 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.500737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.500969 (+ 232us) server_negotiation.cc:407] Connection header received
0927 22:53:55.501189 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.501194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.501270 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.501385 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.501881 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.502609 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.505023 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.506262 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.506387 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.506471 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.506804 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.506936 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:53:55.507421 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.500249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36329 (local address 127.8.204.129:41191)
0927 22:53:55.500857 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.500874 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.501057 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.501399 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.501403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.501426 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.501741 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.501748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.502768 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.502773 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.504868 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.504877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.506467 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.506472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.506648 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.506675 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.506769 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.506883 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":38,"thread_start_us":461,"threads_started":1}
I20260927 22:53:55.508138 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882f22285f27435cb79e539126c5253a" candidate_uuid: "c83995cf1fea46f98178b068465a2b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" is_pre_election: true
I20260927 22:53:55.508404 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882f22285f27435cb79e539126c5253a" candidate_uuid: "c83995cf1fea46f98178b068465a2b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f825266d64599a3147bc88067cbb3" is_pre_election: true
I20260927 22:53:55.508580 16887 raft_consensus.cc:2454] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c83995cf1fea46f98178b068465a2b2a in term 0.
I20260927 22:53:55.508869 17026 raft_consensus.cc:2454] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c83995cf1fea46f98178b068465a2b2a in term 0.
I20260927 22:53:55.509142 16674 leader_election.cc:302] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a, e67022eedb6f407ca25ff12dd73ae2e8; no voters:
I20260927 22:53:55.509502 17104 raft_consensus.cc:2781] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:55.509635 17104 raft_consensus.cc:482] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:55.509744 17104 raft_consensus.cc:3037] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.511900 17104 raft_consensus.cc:504] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.512421 17104 leader_election.cc:288] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 election: Requested vote from peers e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629), eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329)
I20260927 22:53:55.512831 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882f22285f27435cb79e539126c5253a" candidate_uuid: "c83995cf1fea46f98178b068465a2b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
I20260927 22:53:55.512907 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "882f22285f27435cb79e539126c5253a" candidate_uuid: "c83995cf1fea46f98178b068465a2b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f825266d64599a3147bc88067cbb3"
I20260927 22:53:55.513025 16887 raft_consensus.cc:3037] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.513072 17026 raft_consensus.cc:3037] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.516178 17026 raft_consensus.cc:2454] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c83995cf1fea46f98178b068465a2b2a in term 1.
I20260927 22:53:55.516178 16887 raft_consensus.cc:2454] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c83995cf1fea46f98178b068465a2b2a in term 1.
I20260927 22:53:55.516510 16674 leader_election.cc:302] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a, e67022eedb6f407ca25ff12dd73ae2e8; no voters:
I20260927 22:53:55.516748 17104 raft_consensus.cc:2781] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:55.517163 17104 raft_consensus.cc:686] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 1 LEADER]: Becoming Leader. State: Replica: c83995cf1fea46f98178b068465a2b2a, State: Running, Role: LEADER
I20260927 22:53:55.517692 17104 consensus_queue.cc:237] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } }
I20260927 22:53:55.522087 16585 catalog_manager.cc:5495] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c83995cf1fea46f98178b068465a2b2a (127.8.204.129). New cstate: current_term: 1 leader_uuid: "c83995cf1fea46f98178b068465a2b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } health_report { overall_health: HEALTHY } } }
I20260927 22:53:55.538139 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.533361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.8.204.129:42915)
0927 22:53:55.533514 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.533518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.533718 (+ 200us) server_negotiation.cc:407] Connection header received
0927 22:53:55.533865 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.533869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.533947 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.534056 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:55.534338 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.534963 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.536264 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.536625 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.536717 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.537071 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:55.537583 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:55.537694 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.537793 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.537847 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:53:55.545835 17026 raft_consensus.cc:1261] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Refusing update from remote peer c83995cf1fea46f98178b068465a2b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:55.545831 16887 raft_consensus.cc:1261] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Refusing update from remote peer c83995cf1fea46f98178b068465a2b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:55.546814 17104 consensus_queue.cc:1035] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:55.547145 17107 consensus_queue.cc:1035] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:55.558032 17110 mvcc.cc:205] Tried to move back new op lower bound from 7334091307186237440 to 7334091307084341248. Current Snapshot: MvccSnapshot[applied={T|T < 7334091307186237440}]
I20260927 22:53:55.569032 17111 mvcc.cc:205] Tried to move back new op lower bound from 7334091307186237440 to 7334091307084341248. Current Snapshot: MvccSnapshot[applied={T|T < 7334091307186237440 or (T in {7334091307186237440})}]
I20260927 22:53:55.573472 16586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:55.573793 16586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:55.577319 16586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Sending DeleteTablet for 3 replicas of tablet 882f22285f27435cb79e539126c5253a
I20260927 22:53:55.578528 16862 tablet_service.cc:1514] Processing DeleteTablet for tablet 882f22285f27435cb79e539126c5253a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:55 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:50054
I20260927 22:53:55.578763 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet 882f22285f27435cb79e539126c5253a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:55 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:44496
I20260927 22:53:55.579638 17122 tablet_replica.cc:330] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: stopping tablet replica
I20260927 22:53:55.580086 17122 raft_consensus.cc:2229] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:55.580248 16723 tablet_service.cc:1514] Processing DeleteTablet for tablet 882f22285f27435cb79e539126c5253a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:55 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34336
I20260927 22:53:55.580667 17122 raft_consensus.cc:2258] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:55.582857 17122 ts_tablet_manager.cc:1905] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:55.583079 16586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:55.583949 16586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:55.590090 17122 ts_tablet_manager.cc:1918] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:55.590327 17122 log.cc:1196] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/882f22285f27435cb79e539126c5253a
I20260927 22:53:55.590916 17122 ts_tablet_manager.cc:1939] T 882f22285f27435cb79e539126c5253a P eb5f825266d64599a3147bc88067cbb3: Deleting consensus metadata
I20260927 22:53:55.592757 16571 catalog_manager.cc:4841] TS eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329): tablet 882f22285f27435cb79e539126c5253a (table test-table [id=4d393c0f2f974fec9f45b7411824f451]) successfully deleted
I20260927 22:53:55.595785 16861 tablet_service.cc:1467] Processing CreateTablet for tablet 8fbe21a95ebc4b7e8095e1436444d5d5 (DEFAULT_TABLE table=test-table [id=7848c12812d84b0e98a9967e950cb9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:55.595889 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 8fbe21a95ebc4b7e8095e1436444d5d5 (DEFAULT_TABLE table=test-table [id=7848c12812d84b0e98a9967e950cb9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:55.596037 16722 tablet_service.cc:1467] Processing CreateTablet for tablet 8fbe21a95ebc4b7e8095e1436444d5d5 (DEFAULT_TABLE table=test-table [id=7848c12812d84b0e98a9967e950cb9a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:55.596271 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fbe21a95ebc4b7e8095e1436444d5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.596333 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fbe21a95ebc4b7e8095e1436444d5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.596547 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fbe21a95ebc4b7e8095e1436444d5d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.602514 17098 tablet_bootstrap.cc:492] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Bootstrap starting.
I20260927 22:53:55.602651 17096 tablet_bootstrap.cc:492] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Bootstrap starting.
I20260927 22:53:55.604079 17098 tablet_bootstrap.cc:654] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.604537 17096 tablet_bootstrap.cc:654] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.606001 17123 tablet_replica.cc:330] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: stopping tablet replica
I20260927 22:53:55.606541 17123 raft_consensus.cc:2229] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:55.607178 17123 raft_consensus.cc:2258] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:55.607738 17097 tablet_bootstrap.cc:492] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Bootstrap starting.
I20260927 22:53:55.609414 17097 tablet_bootstrap.cc:654] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.610673 17096 tablet_bootstrap.cc:492] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: No bootstrap required, opened a new log
I20260927 22:53:55.610849 17096 ts_tablet_manager.cc:1397] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:53:55.613176 17096 raft_consensus.cc:348] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.613484 17096 raft_consensus.cc:374] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.613601 17096 raft_consensus.cc:729] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67022eedb6f407ca25ff12dd73ae2e8, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.613919 17096 consensus_queue.cc:260] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.614514 17096 ts_tablet_manager.cc:1428] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 22:53:55.615639 17123 ts_tablet_manager.cc:1905] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:55.615875 17097 tablet_bootstrap.cc:492] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: No bootstrap required, opened a new log
I20260927 22:53:55.616025 17097 ts_tablet_manager.cc:1397] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260927 22:53:55.616390 17121 tablet_replica.cc:330] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: stopping tablet replica
I20260927 22:53:55.616999 17121 raft_consensus.cc:2229] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:55.616974 17097 raft_consensus.cc:348] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.617223 17097 raft_consensus.cc:374] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.617324 17097 raft_consensus.cc:729] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83995cf1fea46f98178b068465a2b2a, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.617652 17097 consensus_queue.cc:260] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.618167 17121 raft_consensus.cc:2258] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:55.618880 17097 ts_tablet_manager.cc:1428] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 22:53:55.620079 17098 tablet_bootstrap.cc:492] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: No bootstrap required, opened a new log
I20260927 22:53:55.620266 17098 ts_tablet_manager.cc:1397] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.008s
I20260927 22:53:55.620288 17121 ts_tablet_manager.cc:1905] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:55.621085 17098 raft_consensus.cc:348] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.621361 17098 raft_consensus.cc:374] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.621493 17098 raft_consensus.cc:729] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.621760 17098 consensus_queue.cc:260] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.622402 17098 ts_tablet_manager.cc:1428] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 22:53:55.627725 17121 ts_tablet_manager.cc:1918] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:55.627856 17121 log.cc:1196] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/882f22285f27435cb79e539126c5253a
I20260927 22:53:55.628032 17123 ts_tablet_manager.cc:1918] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:55.628176 17123 log.cc:1196] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/882f22285f27435cb79e539126c5253a
I20260927 22:53:55.628294 17121 ts_tablet_manager.cc:1939] T 882f22285f27435cb79e539126c5253a P e67022eedb6f407ca25ff12dd73ae2e8: Deleting consensus metadata
I20260927 22:53:55.628690 17123 ts_tablet_manager.cc:1939] T 882f22285f27435cb79e539126c5253a P c83995cf1fea46f98178b068465a2b2a: Deleting consensus metadata
I20260927 22:53:55.630192 16570 catalog_manager.cc:4841] TS c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915): tablet 882f22285f27435cb79e539126c5253a (table test-table [id=4d393c0f2f974fec9f45b7411824f451]) successfully deleted
I20260927 22:53:55.630438 16571 catalog_manager.cc:4841] TS e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629): tablet 882f22285f27435cb79e539126c5253a (table test-table [id=4d393c0f2f974fec9f45b7411824f451]) successfully deleted
I20260927 22:53:55.805106 17102 raft_consensus.cc:482] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:55.805343 17102 raft_consensus.cc:504] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.807210 17102 leader_election.cc:288] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329)
I20260927 22:53:55.821693 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.808138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:35291 (local address 127.8.204.131:36329)
0927 22:53:55.808307 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.808313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.813283 (+ 4970us) server_negotiation.cc:407] Connection header received
0927 22:53:55.813558 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.813564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.813641 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.813702 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.814521 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.815677 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.819212 (+ 3535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.820443 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.820615 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.820699 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.821220 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.821363 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:53:55.822402 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.807705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36329 (local address 127.8.204.130:35291)
0927 22:53:55.813169 (+ 5464us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.813188 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.813392 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.813925 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.813931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.813963 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.814339 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.814351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.815882 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.815887 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.818188 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.818203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.820838 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.820844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.821024 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.821061 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.821879 (+ 818us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.822025 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5335,"thread_start_us":5162,"threads_started":1}
I20260927 22:53:55.823216 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fbe21a95ebc4b7e8095e1436444d5d5" candidate_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f825266d64599a3147bc88067cbb3" is_pre_election: true
I20260927 22:53:55.823504 17026 raft_consensus.cc:2454] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e67022eedb6f407ca25ff12dd73ae2e8 in term 0.
I20260927 22:53:55.824030 16814 leader_election.cc:302] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e67022eedb6f407ca25ff12dd73ae2e8, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:55.824465 17102 raft_consensus.cc:2781] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:55.824596 17102 raft_consensus.cc:482] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:55.824681 17102 raft_consensus.cc:3037] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.825863 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.807754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:46939 (local address 127.8.204.129:42915)
0927 22:53:55.807953 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.807966 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.818966 (+ 11000us) server_negotiation.cc:407] Connection header received
0927 22:53:55.819124 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.819132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.819242 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.819394 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.820086 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.821303 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.823607 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.824870 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.825051 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.825124 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.825431 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.825556 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:53:55.826092 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.807722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42915 (local address 127.8.204.130:46939)
0927 22:53:55.818845 (+ 11123us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.818866 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.819006 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.819414 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.819418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.819451 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.819896 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.819908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.821451 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.821455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.823427 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.823439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.825088 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.825094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.825278 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.825299 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.825409 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.825544 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10976,"thread_start_us":10769,"threads_started":1}
I20260927 22:53:55.826962 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fbe21a95ebc4b7e8095e1436444d5d5" candidate_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a" is_pre_election: true
I20260927 22:53:55.827389 16747 raft_consensus.cc:2454] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e67022eedb6f407ca25ff12dd73ae2e8 in term 0.
I20260927 22:53:55.828006 17102 raft_consensus.cc:504] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.828730 17102 leader_election.cc:288] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [CANDIDATE]: Term 1 election: Requested vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329)
I20260927 22:53:55.829125 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fbe21a95ebc4b7e8095e1436444d5d5" candidate_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f825266d64599a3147bc88067cbb3"
I20260927 22:53:55.829123 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fbe21a95ebc4b7e8095e1436444d5d5" candidate_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a"
I20260927 22:53:55.829272 17026 raft_consensus.cc:3037] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.829301 16747 raft_consensus.cc:3037] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.831203 17026 raft_consensus.cc:2454] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e67022eedb6f407ca25ff12dd73ae2e8 in term 1.
I20260927 22:53:55.831565 16814 leader_election.cc:302] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e67022eedb6f407ca25ff12dd73ae2e8, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:55.831832 17102 raft_consensus.cc:2781] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:55.832315 17102 raft_consensus.cc:686] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 LEADER]: Becoming Leader. State: Replica: e67022eedb6f407ca25ff12dd73ae2e8, State: Running, Role: LEADER
I20260927 22:53:55.832568 16747 raft_consensus.cc:2454] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e67022eedb6f407ca25ff12dd73ae2e8 in term 1.
I20260927 22:53:55.832687 17102 consensus_queue.cc:237] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.836294 16585 catalog_manager.cc:5495] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:53:55 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549632, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.190@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:55.927598 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.913536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.8.204.190:44219)
0927 22:53:55.914561 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.914565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.914583 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:53:55.914618 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.914621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.914671 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.914779 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:55.916037 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.916808 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.918013 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.918406 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.918539 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.922170 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:55.922175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:55.922178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:55.922204 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:55.924960 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:55.925624 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:55.925632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:55.925636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:55.925750 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:55.926083 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:55.926087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:55.926091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:55.926362 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:55.926519 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.926935 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.927250 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":911,"thread_start_us":730,"threads_started":1}
I20260927 22:53:55.927601 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.913542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44219 (local address 127.0.0.1:48806)
0927 22:53:55.914225 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.914248 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.914413 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.914798 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.914802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.915605 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:53:55.915865 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.915874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.916925 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.916932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.917865 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.917875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.918577 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.918583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.918826 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.919521 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:53:55.919537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:53:55.921952 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:53:55.925093 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:55.925108 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:55.925116 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:55.925503 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:55.925848 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:53:55.925855 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:53:55.925858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:53:55.925976 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:53:55.926496 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:53:55.926504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:53:55.926684 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.926846 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.927130 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":341,"threads_started":1}
I20260927 22:53:55.933244 16585 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:48806:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:53:55.934245 16585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:55.941239 16722 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa710790b7c4bf2a56a639202b067e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc8b01358d014cd1b740a2b97b6dd2a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:55.941423 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa710790b7c4bf2a56a639202b067e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc8b01358d014cd1b740a2b97b6dd2a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:55.941860 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa710790b7c4bf2a56a639202b067e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.941959 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa710790b7c4bf2a56a639202b067e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.947216 17097 tablet_bootstrap.cc:492] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: Bootstrap starting.
I20260927 22:53:55.947247 16862 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa710790b7c4bf2a56a639202b067e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc8b01358d014cd1b740a2b97b6dd2a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:53:55.947768 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa710790b7c4bf2a56a639202b067e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:55.948997 17097 tablet_bootstrap.cc:654] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.949151 17098 tablet_bootstrap.cc:492] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: Bootstrap starting.
I20260927 22:53:55.950811 17098 tablet_bootstrap.cc:654] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.956477 17096 tablet_bootstrap.cc:492] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: Bootstrap starting.
I20260927 22:53:55.958511 17096 tablet_bootstrap.cc:654] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:55.960445 17098 tablet_bootstrap.cc:492] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: No bootstrap required, opened a new log
I20260927 22:53:55.960606 17098 ts_tablet_manager.cc:1397] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260927 22:53:55.961309 17098 raft_consensus.cc:348] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.962476 17098 raft_consensus.cc:374] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.962570 17098 raft_consensus.cc:729] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.962879 17098 consensus_queue.cc:260] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.963367 17103 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:55.963726 17103 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:55.964210 17098 ts_tablet_manager.cc:1428] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:53:55.966673 17097 tablet_bootstrap.cc:492] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: No bootstrap required, opened a new log
I20260927 22:53:55.966847 17097 ts_tablet_manager.cc:1397] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.016s
I20260927 22:53:55.967679 17097 raft_consensus.cc:348] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.967929 17097 raft_consensus.cc:374] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.968025 17097 raft_consensus.cc:729] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83995cf1fea46f98178b068465a2b2a, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.968262 17097 consensus_queue.cc:260] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.968720 17116 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:55.968952 17097 ts_tablet_manager.cc:1428] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:53:55.969071 17116 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:55.974485 17103 raft_consensus.cc:482] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:55.974658 17096 tablet_bootstrap.cc:492] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: No bootstrap required, opened a new log
I20260927 22:53:55.974653 17103 raft_consensus.cc:504] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.974892 17096 ts_tablet_manager.cc:1397] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260927 22:53:55.975529 17096 raft_consensus.cc:348] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.975731 17096 raft_consensus.cc:374] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:55.975831 17096 raft_consensus.cc:729] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67022eedb6f407ca25ff12dd73ae2e8, State: Initialized, Role: FOLLOWER
I20260927 22:53:55.976054 17096 consensus_queue.cc:260] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.976225 17103 leader_election.cc:288] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:55.976466 17102 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:55.976650 17096 ts_tablet_manager.cc:1428] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:53:55.976751 17102 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:55.984438 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.977585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:55477 (local address 127.8.204.129:42915)
0927 22:53:55.978775 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.978783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.978799 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:53:55.978878 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.978882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.978953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.979029 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.979681 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.980754 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.982433 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.983237 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.983451 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.983525 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.984027 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.984165 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1083}
I20260927 22:53:55.984512 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.976668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42915 (local address 127.8.204.131:55477)
0927 22:53:55.978029 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.978053 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.978188 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.979069 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.979074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.979095 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.979498 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.979508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.980881 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.980887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.982280 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.982291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.983675 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.983681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.983843 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.983875 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.983973 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.984168 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"thread_start_us":1074,"threads_started":1}
I20260927 22:53:55.985198 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa710790b7c4bf2a56a639202b067e2" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a" is_pre_election: true
I20260927 22:53:55.985469 16747 raft_consensus.cc:2454] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 0.
I20260927 22:53:55.985886 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.977302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:43629 (local address 127.8.204.131:57621)
0927 22:53:55.978717 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:55.978733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:55.978881 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:55.979367 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:55.979371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:55.979394 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:53:55.979800 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.979810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.981201 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.981206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.983234 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:55.983249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.984984 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.984990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:55.985289 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.985321 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:53:55.985456 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:53:55.985571 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"thread_start_us":1113,"threads_started":1}
I20260927 22:53:55.985913 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:55.977287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:57621 (local address 127.8.204.130:43629)
0927 22:53:55.977444 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:55.977450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:55.978799 (+ 1349us) server_negotiation.cc:407] Connection header received
0927 22:53:55.979022 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:55.979027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:55.979090 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:55.979176 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:53:55.979968 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.981050 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.983412 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:55.984790 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:55.984950 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:55.985106 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:55.985545 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 22:53:55.985676 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:53:55.985963 16952 leader_election.cc:302] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:55.986342 17103 raft_consensus.cc:2781] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:55.986454 17103 raft_consensus.cc:482] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:55.986585 17103 raft_consensus.cc:3037] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.986548 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa710790b7c4bf2a56a639202b067e2" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" is_pre_election: true
I20260927 22:53:55.986765 16887 raft_consensus.cc:2454] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 0.
I20260927 22:53:55.989501 17103 raft_consensus.cc:504] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.990113 17103 leader_election.cc:288] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 election: Requested vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:55.990734 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa710790b7c4bf2a56a639202b067e2" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a"
I20260927 22:53:55.990955 16747 raft_consensus.cc:3037] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.991338 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa710790b7c4bf2a56a639202b067e2" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
I20260927 22:53:55.991530 16887 raft_consensus.cc:3037] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:55.993387 16887 raft_consensus.cc:2454] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 1.
I20260927 22:53:55.993736 16953 leader_election.cc:302] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e67022eedb6f407ca25ff12dd73ae2e8, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:55.994006 17103 raft_consensus.cc:2781] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:55.994015 16747 raft_consensus.cc:2454] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 1.
I20260927 22:53:55.994517 17103 raft_consensus.cc:686] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [term 1 LEADER]: Becoming Leader. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Running, Role: LEADER
I20260927 22:53:55.994891 17103 consensus_queue.cc:237] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:55.997880 17143 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: TxnStatusTablet state changed. Reason: New leader eb5f825266d64599a3147bc88067cbb3. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:55.998150 17143 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:55.998428 16585 catalog_manager.cc:5495] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb5f825266d64599a3147bc88067cbb3 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } health_report { overall_health: HEALTHY } } }
I20260927 22:53:55.998832 17145 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:53:55.999039 17145 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:53:56.019483 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.013692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36329 (local address 127.0.0.1:44506)
0927 22:53:56.013994 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:56.014009 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:56.014174 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:56.014663 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:56.014668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:56.014680 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:56.014870 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.014887 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.016054 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.016059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.017061 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.017072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.017841 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.017849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.018043 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.018092 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.018891 (+ 799us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.018910 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 22:53:56.019010 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:53:56.019163 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260927 22:53:56.019506 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.013749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.8.204.131:36329)
0927 22:53:56.013926 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.013932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.014086 (+ 154us) server_negotiation.cc:407] Connection header received
0927 22:53:56.014365 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.014371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.014456 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.014646 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:56.015007 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.015900 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.017216 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.017660 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.017794 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.018291 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.018780 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.018938 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.019062 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.019138 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:53:56.023376 16887 raft_consensus.cc:1261] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Refusing update from remote peer eb5f825266d64599a3147bc88067cbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.023820 16747 raft_consensus.cc:1261] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Refusing update from remote peer eb5f825266d64599a3147bc88067cbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.024289 17143 consensus_queue.cc:1035] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.025252 17143 consensus_queue.cc:1035] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.032197 17102 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: TxnStatusTablet state changed. Reason: New leader eb5f825266d64599a3147bc88067cbb3. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:56.032578 17102 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:56.037194 17143 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:56.037415 17143 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:56.038769 17112 mvcc.cc:205] Tried to move back new op lower bound from 7334091309145231360 to 7334091309039050752. Current Snapshot: MvccSnapshot[applied={T|T < 7334091309145231360 or (T in {7334091309145231360})}]
I20260927 22:53:56.043960 17116 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: TxnStatusTablet state changed. Reason: New leader eb5f825266d64599a3147bc88067cbb3. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:56.044215 17116 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:56.044369 17103 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:56.044656 17103 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P eb5f825266d64599a3147bc88067cbb3: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:53:56.047114 17116 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:56.047369 17116 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P c83995cf1fea46f98178b068465a2b2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:56.050508 17102 tablet_replica.cc:439] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } } }
I20260927 22:53:56.050786 17102 tablet_replica.cc:442] T 2aa710790b7c4bf2a56a639202b067e2 P e67022eedb6f407ca25ff12dd73ae2e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:53:56.061636 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.044431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.8.204.130:43629)
0927 22:53:56.044600 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.044606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.055662 (+ 11056us) server_negotiation.cc:407] Connection header received
0927 22:53:56.055891 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.055897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.055999 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.056193 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:56.056705 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.057856 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.059238 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.059903 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.060035 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.060393 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.061084 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.061229 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.061276 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.061354 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:53:56.078027 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.071913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:36083 (local address 127.8.204.131:36329)
0927 22:53:56.072110 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.072120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.072587 (+ 467us) server_negotiation.cc:407] Connection header received
0927 22:53:56.072755 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.072758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.072823 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.072890 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:56.073421 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.074073 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.076050 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.076663 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.076755 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.077328 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.077444 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.077511 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.077711 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.077799 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:53:56.078227 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.071899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36329 (local address 127.8.204.130:36083)
0927 22:53:56.072491 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:56.072509 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:56.072640 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:56.072935 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:56.072939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:56.072951 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:56.073286 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.073297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.074201 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.074208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.075906 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.075921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.076821 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.076830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.077008 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.077078 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.077540 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.077567 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:53:56.077664 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:53:56.077816 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":314,"threads_started":1}
I20260927 22:53:56.087442 16747 raft_consensus.cc:1261] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Refusing update from remote peer e67022eedb6f407ca25ff12dd73ae2e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.087543 17026 raft_consensus.cc:1261] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Refusing update from remote peer e67022eedb6f407ca25ff12dd73ae2e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.088003 17129 consensus_queue.cc:1035] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.088438 17102 consensus_queue.cc:1035] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.133383 17091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.123212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:55805 (local address 127.8.204.130:43629)
0927 22:53:56.123396 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.123403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.127815 (+ 4412us) server_negotiation.cc:407] Connection header received
0927 22:53:56.128023 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.128028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.128099 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.128188 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:56.128929 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.129921 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.131589 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.132021 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.132172 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.132626 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.132742 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.132838 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.132984 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.133076 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:53:56.134313 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.123240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:43629 (local address 127.8.204.131:55805)
0927 22:53:56.127719 (+ 4479us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:56.127735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:56.127887 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:56.128347 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:56.128353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:56.128364 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:56.128753 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.128767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.130095 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.130098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.131447 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.131462 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.132200 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.132208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.132396 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.132458 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.132839 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.132852 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:53:56.133700 (+ 848us) client_negotiation.cc:240] Negotiation successful
0927 22:53:56.133882 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4317,"thread_start_us":4185,"threads_started":1}
I20260927 22:53:56.167894 16586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:56.168159 16586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:56.171478 16586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Sending DeleteTablet for 3 replicas of tablet 8fbe21a95ebc4b7e8095e1436444d5d5
I20260927 22:53:56.172178 16722 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fbe21a95ebc4b7e8095e1436444d5d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34336
I20260927 22:53:56.172446 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fbe21a95ebc4b7e8095e1436444d5d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:44496
I20260927 22:53:56.173591 16862 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fbe21a95ebc4b7e8095e1436444d5d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:50054
I20260927 22:53:56.206102 17166 tablet_replica.cc:330] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: stopping tablet replica
I20260927 22:53:56.207827 17166 raft_consensus.cc:2229] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:56.208638 17166 raft_consensus.cc:2258] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.210043 17166 ts_tablet_manager.cc:1905] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.210620 17165 tablet_replica.cc:330] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: stopping tablet replica
I20260927 22:53:56.215473 17165 raft_consensus.cc:2229] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:56.215849 17165 raft_consensus.cc:2258] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.216058 17164 tablet_replica.cc:330] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: stopping tablet replica
I20260927 22:53:56.216257 17164 raft_consensus.cc:2229] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:56.216460 17164 raft_consensus.cc:2258] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.217228 17165 ts_tablet_manager.cc:1905] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.217468 17164 ts_tablet_manager.cc:1905] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.224972 17164 ts_tablet_manager.cc:1918] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:56.225142 17164 log.cc:1196] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/8fbe21a95ebc4b7e8095e1436444d5d5
I20260927 22:53:56.225593 17164 ts_tablet_manager.cc:1939] T 8fbe21a95ebc4b7e8095e1436444d5d5 P c83995cf1fea46f98178b068465a2b2a: Deleting consensus metadata
I20260927 22:53:56.226805 16570 catalog_manager.cc:4841] TS c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915): tablet 8fbe21a95ebc4b7e8095e1436444d5d5 (table test-table [id=7848c12812d84b0e98a9967e950cb9a5]) successfully deleted
I20260927 22:53:56.228170 17166 ts_tablet_manager.cc:1918] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:56.228184 17165 ts_tablet_manager.cc:1918] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:56.228338 17165 log.cc:1196] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/8fbe21a95ebc4b7e8095e1436444d5d5
I20260927 22:53:56.228479 17166 log.cc:1196] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/8fbe21a95ebc4b7e8095e1436444d5d5
I20260927 22:53:56.228821 17165 ts_tablet_manager.cc:1939] T 8fbe21a95ebc4b7e8095e1436444d5d5 P eb5f825266d64599a3147bc88067cbb3: Deleting consensus metadata
I20260927 22:53:56.229053 17166 ts_tablet_manager.cc:1939] T 8fbe21a95ebc4b7e8095e1436444d5d5 P e67022eedb6f407ca25ff12dd73ae2e8: Deleting consensus metadata
I20260927 22:53:56.229987 16571 catalog_manager.cc:4841] TS eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329): tablet 8fbe21a95ebc4b7e8095e1436444d5d5 (table test-table [id=7848c12812d84b0e98a9967e950cb9a5]) successfully deleted
I20260927 22:53:56.230247 16571 catalog_manager.cc:4841] TS e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629): tablet 8fbe21a95ebc4b7e8095e1436444d5d5 (table test-table [id=7848c12812d84b0e98a9967e950cb9a5]) successfully deleted
Sep 27 22:53:56 dist-test-slave-40vx krb5kdc[16541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:53:56 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:53:56 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:53:56.317211 17137 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:53:56.296630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.8.204.190:44219)
0927 22:53:56.296858 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.296865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.304241 (+ 7376us) server_negotiation.cc:407] Connection header received
0927 22:53:56.304412 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.304419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.304508 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.304627 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:56.306306 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.307494 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.309337 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.310002 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.310155 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.316871 (+ 6716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48818: BlockingRecv error: recv got EOF from 127.0.0.1:48818 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 22:53:56 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM
I20260927 22:53:56.337644 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.320967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.8.204.190:44219)
0927 22:53:56.321160 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.321166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.321200 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:53:56.321375 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.321382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.321462 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.321530 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:56.323199 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.324323 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.325997 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.326668 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.326795 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.331194 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:56.331201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:56.331207 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:56.331249 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:56.334754 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:56.335581 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:56.335587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:56.335592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:56.335738 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:56.336144 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:56.336150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:56.336153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:56.336494 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:56.336763 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.337047 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.337297 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:53:56.341467 16586 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:53:56.342186 16586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:56.350188 16862 tablet_service.cc:1467] Processing CreateTablet for tablet 3362603f98db4b73a480e6dc6cc81d49 (DEFAULT_TABLE table=test-table [id=91fec8044351454085f2c43f8ab4870a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:56.350628 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3362603f98db4b73a480e6dc6cc81d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:56.350561 17001 tablet_service.cc:1467] Processing CreateTablet for tablet 3362603f98db4b73a480e6dc6cc81d49 (DEFAULT_TABLE table=test-table [id=91fec8044351454085f2c43f8ab4870a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:56.350948 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3362603f98db4b73a480e6dc6cc81d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:56.355532 16722 tablet_service.cc:1467] Processing CreateTablet for tablet 3362603f98db4b73a480e6dc6cc81d49 (DEFAULT_TABLE table=test-table [id=91fec8044351454085f2c43f8ab4870a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:56.355679 17098 tablet_bootstrap.cc:492] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Bootstrap starting.
I20260927 22:53:56.355976 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3362603f98db4b73a480e6dc6cc81d49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:56.357344 17098 tablet_bootstrap.cc:654] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:56.362658 17097 tablet_bootstrap.cc:492] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Bootstrap starting.
I20260927 22:53:56.364187 17097 tablet_bootstrap.cc:654] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:56.365098 17096 tablet_bootstrap.cc:492] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Bootstrap starting.
I20260927 22:53:56.366592 17096 tablet_bootstrap.cc:654] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:56.368673 17096 tablet_bootstrap.cc:492] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: No bootstrap required, opened a new log
I20260927 22:53:56.368865 17096 ts_tablet_manager.cc:1397] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:53:56.369725 17096 raft_consensus.cc:348] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.370090 17096 raft_consensus.cc:374] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:56.370208 17096 raft_consensus.cc:729] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67022eedb6f407ca25ff12dd73ae2e8, State: Initialized, Role: FOLLOWER
I20260927 22:53:56.370441 17096 consensus_queue.cc:260] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.371660 17096 ts_tablet_manager.cc:1428] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:53:56.372488 17098 tablet_bootstrap.cc:492] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: No bootstrap required, opened a new log
I20260927 22:53:56.372689 17098 ts_tablet_manager.cc:1397] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20260927 22:53:56.374024 17098 raft_consensus.cc:348] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.374367 17098 raft_consensus.cc:374] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:56.374518 17098 raft_consensus.cc:729] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Initialized, Role: FOLLOWER
I20260927 22:53:56.374822 17098 consensus_queue.cc:260] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.377539 17098 ts_tablet_manager.cc:1428] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260927 22:53:56.378975 17097 tablet_bootstrap.cc:492] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: No bootstrap required, opened a new log
I20260927 22:53:56.379138 17097 ts_tablet_manager.cc:1397] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20260927 22:53:56.379925 17097 raft_consensus.cc:348] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.380148 17097 raft_consensus.cc:374] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:56.380245 17097 raft_consensus.cc:729] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83995cf1fea46f98178b068465a2b2a, State: Initialized, Role: FOLLOWER
I20260927 22:53:56.380486 17097 consensus_queue.cc:260] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.381409 17097 ts_tablet_manager.cc:1428] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:53:56.538635 17155 raft_consensus.cc:482] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:56.538893 17155 raft_consensus.cc:504] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.539752 17155 leader_election.cc:288] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:56.540367 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362603f98db4b73a480e6dc6cc81d49" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a" is_pre_election: true
I20260927 22:53:56.540453 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362603f98db4b73a480e6dc6cc81d49" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" is_pre_election: true
I20260927 22:53:56.540645 16747 raft_consensus.cc:2454] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 0.
I20260927 22:53:56.540716 16887 raft_consensus.cc:2454] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 0.
I20260927 22:53:56.541205 16953 leader_election.cc:302] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e67022eedb6f407ca25ff12dd73ae2e8, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:56.541612 17155 raft_consensus.cc:2781] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:56.541724 17155 raft_consensus.cc:482] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:56.541811 17155 raft_consensus.cc:3037] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:56.545145 17155 raft_consensus.cc:504] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.546197 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362603f98db4b73a480e6dc6cc81d49" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a"
I20260927 22:53:56.546253 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3362603f98db4b73a480e6dc6cc81d49" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
I20260927 22:53:56.546414 16747 raft_consensus.cc:3037] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:56.546516 16887 raft_consensus.cc:3037] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:56.549202 16747 raft_consensus.cc:2454] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 1.
I20260927 22:53:56.549618 16887 raft_consensus.cc:2454] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 1.
I20260927 22:53:56.549727 16952 leader_election.cc:302] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:56.550033 17143 raft_consensus.cc:2781] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:56.550171 17143 raft_consensus.cc:686] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 1 LEADER]: Becoming Leader. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Running, Role: LEADER
I20260927 22:53:56.550484 17143 consensus_queue.cc:237] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.550171 17155 leader_election.cc:288] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 election: Requested vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:56.553426 16585 catalog_manager.cc:5495] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb5f825266d64599a3147bc88067cbb3 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } health_report { overall_health: HEALTHY } } }
I20260927 22:53:56.598469 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.593609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.8.204.131:36329)
0927 22:53:56.594025 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.594029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.594044 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:53:56.594103 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.594106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.594165 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.594280 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:56.594638 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.595416 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.596727 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.597202 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.597333 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.597727 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.597872 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.597953 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.598099 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.598171 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":233,"threads_started":1}
I20260927 22:53:56.601648 16887 raft_consensus.cc:1261] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Refusing update from remote peer eb5f825266d64599a3147bc88067cbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.601722 16747 raft_consensus.cc:1261] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Refusing update from remote peer eb5f825266d64599a3147bc88067cbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.602274 17155 consensus_queue.cc:1035] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.602538 17143 consensus_queue.cc:1035] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.623129 16585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:56.623412 16585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:56.632473 16585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Sending DeleteTablet for 3 replicas of tablet 3362603f98db4b73a480e6dc6cc81d49
I20260927 22:53:56.633679 16722 tablet_service.cc:1514] Processing DeleteTablet for tablet 3362603f98db4b73a480e6dc6cc81d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34336
I20260927 22:53:56.633988 17164 tablet_replica.cc:330] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: stopping tablet replica
I20260927 22:53:56.633996 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet 3362603f98db4b73a480e6dc6cc81d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:44496
I20260927 22:53:56.634219 17164 raft_consensus.cc:2229] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:56.634330 16862 tablet_service.cc:1514] Processing DeleteTablet for tablet 3362603f98db4b73a480e6dc6cc81d49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:50054
I20260927 22:53:56.634439 17164 raft_consensus.cc:2258] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.634578 17166 tablet_replica.cc:330] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: stopping tablet replica
I20260927 22:53:56.634696 17165 tablet_replica.cc:330] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: stopping tablet replica
I20260927 22:53:56.634792 17166 raft_consensus.cc:2229] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:56.634871 17165 raft_consensus.cc:2229] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:56.635010 17166 raft_consensus.cc:2258] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.635406 17165 raft_consensus.cc:2258] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.636144 16585 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:53:56.636596 17165 ts_tablet_manager.cc:1905] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 22:53:56.636973 16585 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:53:56.638794 17166 ts_tablet_manager.cc:1905] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.643941 17165 ts_tablet_manager.cc:1918] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:56.644213 17165 log.cc:1196] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/3362603f98db4b73a480e6dc6cc81d49
I20260927 22:53:56.644742 17164 ts_tablet_manager.cc:1905] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.644771 17165 ts_tablet_manager.cc:1939] T 3362603f98db4b73a480e6dc6cc81d49 P eb5f825266d64599a3147bc88067cbb3: Deleting consensus metadata
I20260927 22:53:56.646378 17166 ts_tablet_manager.cc:1918] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:56.646559 17166 log.cc:1196] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/3362603f98db4b73a480e6dc6cc81d49
I20260927 22:53:56.647068 17166 ts_tablet_manager.cc:1939] T 3362603f98db4b73a480e6dc6cc81d49 P e67022eedb6f407ca25ff12dd73ae2e8: Deleting consensus metadata
I20260927 22:53:56.647960 16571 catalog_manager.cc:4841] TS eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329): tablet 3362603f98db4b73a480e6dc6cc81d49 (table test-table [id=91fec8044351454085f2c43f8ab4870a]) successfully deleted
I20260927 22:53:56.651137 17164 ts_tablet_manager.cc:1918] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:53:56.651381 17164 log.cc:1196] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/3362603f98db4b73a480e6dc6cc81d49
I20260927 22:53:56.651863 17164 ts_tablet_manager.cc:1939] T 3362603f98db4b73a480e6dc6cc81d49 P c83995cf1fea46f98178b068465a2b2a: Deleting consensus metadata
I20260927 22:53:56.652520 16571 catalog_manager.cc:4841] TS e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629): tablet 3362603f98db4b73a480e6dc6cc81d49 (table test-table [id=91fec8044351454085f2c43f8ab4870a]) successfully deleted
I20260927 22:53:56.653004 16570 catalog_manager.cc:4841] TS c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915): tablet 3362603f98db4b73a480e6dc6cc81d49 (table test-table [id=91fec8044351454085f2c43f8ab4870a]) successfully deleted
I20260927 22:53:56.656922 16722 tablet_service.cc:1467] Processing CreateTablet for tablet f97593d609694872b16891e2addcdb4d (DEFAULT_TABLE table=test-table [id=bd2ca878bde0467eadf555e03365b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:56.657323 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97593d609694872b16891e2addcdb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:56.657521 16862 tablet_service.cc:1467] Processing CreateTablet for tablet f97593d609694872b16891e2addcdb4d (DEFAULT_TABLE table=test-table [id=bd2ca878bde0467eadf555e03365b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:56.657932 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97593d609694872b16891e2addcdb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:56.663607 17097 tablet_bootstrap.cc:492] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Bootstrap starting.
I20260927 22:53:56.663641 17001 tablet_service.cc:1467] Processing CreateTablet for tablet f97593d609694872b16891e2addcdb4d (DEFAULT_TABLE table=test-table [id=bd2ca878bde0467eadf555e03365b522]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:53:56.664067 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f97593d609694872b16891e2addcdb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:53:56.665184 17097 tablet_bootstrap.cc:654] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:56.676066 17097 tablet_bootstrap.cc:492] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: No bootstrap required, opened a new log
I20260927 22:53:56.676225 17096 tablet_bootstrap.cc:492] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Bootstrap starting.
I20260927 22:53:56.676373 17097 ts_tablet_manager.cc:1397] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20260927 22:53:56.677191 17097 raft_consensus.cc:348] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.677479 17097 raft_consensus.cc:374] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:56.677605 17097 raft_consensus.cc:729] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83995cf1fea46f98178b068465a2b2a, State: Initialized, Role: FOLLOWER
I20260927 22:53:56.677850 17097 consensus_queue.cc:260] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.678092 17096 tablet_bootstrap.cc:654] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Neither blocks nor log segments found. Creating new log.
I20260927 22:53:56.681337 17097 ts_tablet_manager.cc:1428] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260927 22:53:56.681900 17098 tablet_bootstrap.cc:492] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Bootstrap starting.
I20260927 22:53:56.682160 17116 raft_consensus.cc:482] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:56.682286 17116 raft_consensus.cc:504] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.682828 17116 leader_election.cc:288] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629), eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329)
I20260927 22:53:56.683485 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97593d609694872b16891e2addcdb4d" candidate_uuid: "c83995cf1fea46f98178b068465a2b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb5f825266d64599a3147bc88067cbb3" is_pre_election: true
I20260927 22:53:56.683646 17098 tablet_bootstrap.cc:654] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Neither blocks nor log segments found. Creating new log.
W20260927 22:53:56.684593 16674 leader_election.cc:341] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329): Illegal state: must be running to vote when last-logged opid is not known
I20260927 22:53:56.687551 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97593d609694872b16891e2addcdb4d" candidate_uuid: "c83995cf1fea46f98178b068465a2b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" is_pre_election: true
W20260927 22:53:56.688202 16674 leader_election.cc:341] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629): Illegal state: must be running to vote when last-logged opid is not known
I20260927 22:53:56.688388 16674 leader_election.cc:302] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a; no voters: e67022eedb6f407ca25ff12dd73ae2e8, eb5f825266d64599a3147bc88067cbb3
I20260927 22:53:56.688853 17116 raft_consensus.cc:2726] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:53:56.694734 17098 tablet_bootstrap.cc:492] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: No bootstrap required, opened a new log
I20260927 22:53:56.695003 17098 ts_tablet_manager.cc:1397] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20260927 22:53:56.696030 17096 tablet_bootstrap.cc:492] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: No bootstrap required, opened a new log
I20260927 22:53:56.695858 17098 raft_consensus.cc:348] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.696147 17098 raft_consensus.cc:374] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:56.696277 17098 raft_consensus.cc:729] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Initialized, Role: FOLLOWER
I20260927 22:53:56.696303 17096 ts_tablet_manager.cc:1397] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.001s
I20260927 22:53:56.696638 17098 consensus_queue.cc:260] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.698314 17096 raft_consensus.cc:348] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.698684 17096 raft_consensus.cc:374] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:53:56.698824 17096 raft_consensus.cc:729] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67022eedb6f407ca25ff12dd73ae2e8, State: Initialized, Role: FOLLOWER
I20260927 22:53:56.699167 17096 consensus_queue.cc:260] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.699735 17098 ts_tablet_manager.cc:1428] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:53:56.701282 17096 ts_tablet_manager.cc:1428] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:53:56.827498 17155 raft_consensus.cc:482] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:53:56.827764 17155 raft_consensus.cc:504] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.828510 17155 leader_election.cc:288] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:56.828981 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97593d609694872b16891e2addcdb4d" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a" is_pre_election: true
I20260927 22:53:56.829238 16747 raft_consensus.cc:2454] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 0.
I20260927 22:53:56.829257 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97593d609694872b16891e2addcdb4d" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" is_pre_election: true
I20260927 22:53:56.829502 16887 raft_consensus.cc:2454] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 0.
I20260927 22:53:56.829768 16952 leader_election.cc:302] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:56.830081 17155 raft_consensus.cc:2781] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:53:56.830188 17155 raft_consensus.cc:482] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:53:56.830364 17155 raft_consensus.cc:3037] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:56.832585 17155 raft_consensus.cc:504] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.833323 17155 leader_election.cc:288] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 election: Requested vote from peers c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915), e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629)
I20260927 22:53:56.833683 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97593d609694872b16891e2addcdb4d" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c83995cf1fea46f98178b068465a2b2a"
I20260927 22:53:56.833897 16747 raft_consensus.cc:3037] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:56.833999 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f97593d609694872b16891e2addcdb4d" candidate_uuid: "eb5f825266d64599a3147bc88067cbb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e67022eedb6f407ca25ff12dd73ae2e8"
I20260927 22:53:56.834228 16887 raft_consensus.cc:3037] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:53:56.836912 16747 raft_consensus.cc:2454] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 1.
I20260927 22:53:56.837141 16887 raft_consensus.cc:2454] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb5f825266d64599a3147bc88067cbb3 in term 1.
I20260927 22:53:56.837348 16952 leader_election.cc:302] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c83995cf1fea46f98178b068465a2b2a, eb5f825266d64599a3147bc88067cbb3; no voters:
I20260927 22:53:56.837620 17155 raft_consensus.cc:2781] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:53:56.837826 17155 raft_consensus.cc:686] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 1 LEADER]: Becoming Leader. State: Replica: eb5f825266d64599a3147bc88067cbb3, State: Running, Role: LEADER
I20260927 22:53:56.838104 17155 consensus_queue.cc:237] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } }
I20260927 22:53:56.841056 16586 catalog_manager.cc:5495] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb5f825266d64599a3147bc88067cbb3 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "eb5f825266d64599a3147bc88067cbb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb5f825266d64599a3147bc88067cbb3" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36329 } health_report { overall_health: HEALTHY } } }
I20260927 22:53:56.903131 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.885269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:42237 (local address 127.8.204.131:36329)
0927 22:53:56.885461 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.885468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.897461 (+ 11993us) server_negotiation.cc:407] Connection header received
0927 22:53:56.897657 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.897661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.897723 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.897805 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:53:56.898413 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.899332 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.901440 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.901878 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.901975 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.902464 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.902566 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.902637 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.902794 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.902874 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:53:56.903540 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.885106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36329 (local address 127.8.204.131:42237)
0927 22:53:56.897337 (+ 12231us) negotiation.cc:107] Waiting for socket to connect
0927 22:53:56.897360 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:53:56.897534 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:53:56.897876 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:53:56.897884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:53:56.897900 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:53:56.898262 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.898273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.899471 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.899476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.901278 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:53:56.901291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.902065 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.902071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:53:56.902267 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.902318 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:53:56.902674 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:53:56.902685 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:53:56.902773 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:53:56.902968 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12034,"mutex_wait_us":28,"thread_start_us":11780,"threads_started":1}
I20260927 22:53:56.912176 16887 raft_consensus.cc:1261] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Refusing update from remote peer eb5f825266d64599a3147bc88067cbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.912671 16747 raft_consensus.cc:1261] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Refusing update from remote peer eb5f825266d64599a3147bc88067cbb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:53:56.912779 17155 consensus_queue.cc:1035] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e67022eedb6f407ca25ff12dd73ae2e8" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.913537 17155 consensus_queue.cc:1035] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c83995cf1fea46f98178b068465a2b2a" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:53:56.963440 16586 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:56.963712 16586 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:53:56.966967 16586 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a831b2e61e142648ee6e8abed13a6d5: Sending DeleteTablet for 3 replicas of tablet f97593d609694872b16891e2addcdb4d
I20260927 22:53:56.967819 16722 tablet_service.cc:1514] Processing DeleteTablet for tablet f97593d609694872b16891e2addcdb4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34336
I20260927 22:53:56.967885 17001 tablet_service.cc:1514] Processing DeleteTablet for tablet f97593d609694872b16891e2addcdb4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:44496
I20260927 22:53:56.968117 17164 tablet_replica.cc:330] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: stopping tablet replica
I20260927 22:53:56.968170 17165 tablet_replica.cc:330] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: stopping tablet replica
I20260927 22:53:56.968295 17164 raft_consensus.cc:2229] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:56.968346 17165 raft_consensus.cc:2229] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:53:56.968505 17164 raft_consensus.cc:2258] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.968665 17165 raft_consensus.cc:2258] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.969300 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 16657
I20260927 22:53:56.969318 16862 tablet_service.cc:1514] Processing DeleteTablet for tablet f97593d609694872b16891e2addcdb4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:53:56 UTC) from {username='oryx', principal='oryx/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:50054
I20260927 22:53:56.969664 17166 tablet_replica.cc:330] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: stopping tablet replica
I20260927 22:53:56.969837 17166 raft_consensus.cc:2229] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:53:56.970114 17166 raft_consensus.cc:2258] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:53:56.971655 17166 ts_tablet_manager.cc:1905] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.971678 17165 ts_tablet_manager.cc:1905] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.976384 17164 ts_tablet_manager.cc:1905] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:53:56.979688 17166 ts_tablet_manager.cc:1918] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:56.979861 17166 log.cc:1196] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/f97593d609694872b16891e2addcdb4d
I20260927 22:53:56.980377 17166 ts_tablet_manager.cc:1939] T f97593d609694872b16891e2addcdb4d P e67022eedb6f407ca25ff12dd73ae2e8: Deleting consensus metadata
I20260927 22:53:56.981104 17165 ts_tablet_manager.cc:1918] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:56.981458 16571 catalog_manager.cc:4841] TS e67022eedb6f407ca25ff12dd73ae2e8 (127.8.204.130:43629): tablet f97593d609694872b16891e2addcdb4d (table test-table [id=bd2ca878bde0467eadf555e03365b522]) successfully deleted
I20260927 22:53:56.982127 17165 log.cc:1196] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/f97593d609694872b16891e2addcdb4d
I20260927 22:53:56.982606 17165 ts_tablet_manager.cc:1939] T f97593d609694872b16891e2addcdb4d P eb5f825266d64599a3147bc88067cbb3: Deleting consensus metadata
I20260927 22:53:56.983956 16571 catalog_manager.cc:4841] TS eb5f825266d64599a3147bc88067cbb3 (127.8.204.131:36329): tablet f97593d609694872b16891e2addcdb4d (table test-table [id=bd2ca878bde0467eadf555e03365b522]) successfully deleted
I20260927 22:53:56.984469 17164 ts_tablet_manager.cc:1918] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:53:56.984613 17164 log.cc:1196] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/f97593d609694872b16891e2addcdb4d
I20260927 22:53:56.985071 17164 ts_tablet_manager.cc:1939] T f97593d609694872b16891e2addcdb4d P c83995cf1fea46f98178b068465a2b2a: Deleting consensus metadata
I20260927 22:53:56.986138 16570 catalog_manager.cc:4841] TS c83995cf1fea46f98178b068465a2b2a (127.8.204.129:42915): tablet f97593d609694872b16891e2addcdb4d (table test-table [id=bd2ca878bde0467eadf555e03365b522]) successfully deleted
Sep 27 22:53:56 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.129@KRBTEST.COM
I20260927 22:53:56.999135 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:56.970622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34350 (local address 127.8.204.129:42915)
0927 22:53:56.982331 (+ 11709us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:56.982339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:56.984178 (+ 1839us) server_negotiation.cc:407] Connection header received
0927 22:53:56.984394 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:56.984400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:56.984506 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:56.984625 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:56.986412 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.987102 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.988353 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:56.988851 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:56.988990 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:56.993126 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:56.993140 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:56.993155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:56.993186 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:56.996246 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:56.996937 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:56.996944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:56.996946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:56.997155 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:56.997548 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:56.997552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:56.997556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:56.997877 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:56.998230 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:56.998576 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 22:53:56.998794 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11463,"thread_start_us":11320,"threads_started":1}
I20260927 22:53:56.999863 16788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:57.137390 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 16657
I20260927 22:53:57.159576 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 16797
Sep 27 22:53:57 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.130@KRBTEST.COM
I20260927 22:53:57.173727 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:57.160306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50072 (local address 127.8.204.130:43629)
0927 22:53:57.160833 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:57.160837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:57.160853 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:53:57.160906 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:57.160910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:57.160971 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:57.161065 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:57.162140 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:57.162858 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:57.164194 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:57.164580 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:57.164674 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:57.168577 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:57.168589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:57.168604 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:57.168633 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:57.171388 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:57.171963 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:57.171969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:57.171972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:57.172081 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:57.172378 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:57.172381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:57.172383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:57.172724 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:57.172932 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:57.173178 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:53:57.173338 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":302,"threads_started":1}
I20260927 22:53:57.174350 16927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:57.296483 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 16797
I20260927 22:53:57.318351 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 16936
Sep 27 22:53:57 dist-test-slave-40vx krb5kdc[16541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.204.131@KRBTEST.COM
I20260927 22:53:57.333127 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:53:57.319195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.8.204.131:36329)
0927 22:53:57.319390 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:53:57.319396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:53:57.319415 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:53:57.319590 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:53:57.319594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:53:57.319684 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:53:57.319804 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:53:57.321123 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:57.321831 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:57.323070 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:53:57.323614 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:53:57.323748 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:53:57.327972 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:53:57.327986 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:53:57.328003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:53:57.328048 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:53:57.330805 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:57.331471 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:57.331478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:57.331481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:57.331629 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:53:57.331964 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:53:57.331967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:53:57.331969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:53:57.332271 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:53:57.332467 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:53:57.332731 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:53:57.332880 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:53:57.333814 17066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:57.477449 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 16936
I20260927 22:53:57.502259 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 16554
I20260927 22:53:57.503465 16626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:53:57.617298 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 16554
2026-09-27T22:53:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6719 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:57 dist-test-slave-40vx krb5kdc[17207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:57 dist-test-slave-40vx krb5kdc[17207](info): set up 2 sockets
Sep 27 22:53:57 dist-test-slave-40vx krb5kdc[17207](info): commencing operation
krb5kdc: starting...
W20260927 22:53:58.704252 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:53:58 dist-test-slave-40vx krb5kdc[17207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:53:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:53:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:53:58.916530 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:46325
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:46325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:53:59.237180 17220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:53:59.237483 17220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:53:59.237583 17220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:53:59.245234 17220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:53:59.245333 17220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:53:59.245389 17220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:53:59.245445 17220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:53:59.245496 17220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:53:59.255251 17220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:46325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:46325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:53:59.256577 17220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:53:59.257874 17220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:53:59.270129 17228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:59.270102 17226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:53:59.270076 17225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:53:59.270892 17220 server_base.cc:1034] running on GCE node
I20260927 22:53:59.271746 17220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:53:59.273156 17220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:53:59.274348 17220 hybrid_clock.cc:648] HybridClock initialized: now 1790549639274298 us; error 41 us; skew 500 ppm
I20260927 22:53:59.274708 17220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:53:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1724 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:53:59 dist-test-slave-40vx krb5kdc[17234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:53:59 dist-test-slave-40vx krb5kdc[17234](info): set up 2 sockets
Sep 27 22:53:59 dist-test-slave-40vx krb5kdc[17234](info): commencing operation
krb5kdc: starting...
W20260927 22:54:00.436393 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:54:00 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:54:00.639919 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45041
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:45041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:00.971010 17247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:00.971284 17247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:00.971467 17247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:00.978449 17247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:00.978545 17247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:00.978595 17247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:00.978672 17247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:00.978760 17247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:00.988561 17247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:45041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:00.989779 17247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:00.991055 17247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:01.002424 17252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:01.002380 17255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:01.002475 17253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:01.003235 17247 server_base.cc:1034] running on GCE node
I20260927 22:54:01.003996 17247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:01.005314 17247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:01.006536 17247 hybrid_clock.cc:648] HybridClock initialized: now 1790549641006500 us; error 56 us; skew 500 ppm
I20260927 22:54:01.006873 17247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:01 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:01.009954 17247 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:54:01.011340 17247 webserver.cc:460] Webserver started at http://127.8.204.190:40873/ using document root <none> and password file <none>
I20260927 22:54:01.011979 17247 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:01.012133 17247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:01.012426 17247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:01.015194 17247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "d62672e841184c8e992effaa48f19dce"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "7ae5c98af33042591d208164d7d2bbaf"
server_key_iv: "e08afb904d792b1a6504ba05c6381d54"
server_key_version: "encryptionkey@0"
I20260927 22:54:01.015996 17247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "d62672e841184c8e992effaa48f19dce"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "7ae5c98af33042591d208164d7d2bbaf"
server_key_iv: "e08afb904d792b1a6504ba05c6381d54"
server_key_version: "encryptionkey@0"
I20260927 22:54:01.020768 17247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:54:01.024147 17262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:01.025336 17247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260927 22:54:01.025507 17247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "d62672e841184c8e992effaa48f19dce"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "7ae5c98af33042591d208164d7d2bbaf"
server_key_iv: "e08afb904d792b1a6504ba05c6381d54"
server_key_version: "encryptionkey@0"
I20260927 22:54:01.025714 17247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:01.038892 17247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:01.042846 17247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:01.043118 17247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:01.059725 17247 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:45041
I20260927 22:54:01.059736 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:45041 every 8 connection(s)
I20260927 22:54:01.060961 17247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:01.064364 17314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:01.068430 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17247
I20260927 22:54:01.068647 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:54:01.069262 9010 external_mini_cluster.cc:1428] Setting key 50cfe3a0d91a6873370aab4efdf89185
I20260927 22:54:01.074690 17314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Bootstrap starting.
I20260927 22:54:01.078088 17314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:01.079233 17314 log.cc:824] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:01.082472 17314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: No bootstrap required, opened a new log
Sep 27 22:54:01 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:01.090044 17314 raft_consensus.cc:348] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:01.090487 17314 raft_consensus.cc:374] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:01.090591 17314 raft_consensus.cc:729] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62672e841184c8e992effaa48f19dce, State: Initialized, Role: FOLLOWER
I20260927 22:54:01.091181 17314 consensus_queue.cc:260] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:01.091468 17314 raft_consensus.cc:388] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:01.091522 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:01.071056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.8.204.190:45041)
0927 22:54:01.071869 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:01.071880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:01.071917 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:54:01.073607 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:01.073626 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:01.073969 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:01.074263 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:01.075971 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:01.077260 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:01.079009 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:01.079841 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:01.080027 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:01.084287 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:01.084294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:01.084310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:01.084341 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:01.087434 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:01.088107 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:01.088113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:01.088118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:01.088262 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:01.088666 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:01.088670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:01.088672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:01.089162 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:01.089449 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:01.089877 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 22:54:01.090268 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":434,"threads_started":1}
I20260927 22:54:01.091634 17314 raft_consensus.cc:482] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:01.091822 17314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:01.094060 17314 raft_consensus.cc:504] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:01.094513 17314 leader_election.cc:302] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d62672e841184c8e992effaa48f19dce; no voters:
I20260927 22:54:01.095039 17314 leader_election.cc:288] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:01.095170 17319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:01.095885 17319 raft_consensus.cc:686] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 LEADER]: Becoming Leader. State: Replica: d62672e841184c8e992effaa48f19dce, State: Running, Role: LEADER
I20260927 22:54:01.096403 17319 consensus_queue.cc:237] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:01.096928 17314 sys_catalog.cc:564] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:01.098963 17320 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d62672e841184c8e992effaa48f19dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } } }
I20260927 22:54:01.099375 17320 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:01.098984 17321 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d62672e841184c8e992effaa48f19dce. Latest consensus state: current_term: 1 leader_uuid: "d62672e841184c8e992effaa48f19dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } } }
I20260927 22:54:01.100180 17321 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:01.101711 17324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:01.106207 17324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:01.113855 17324 catalog_manager.cc:1347] Generated new cluster ID: aeb291d0a07b4680ae168d600fbfaa17
I20260927 22:54:01.114029 17324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:01.123809 17324 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:01.125412 17324 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:01.132041 17324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Generated new TSK 0
I20260927 22:54:01.132886 17324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:54:01.186661 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45041
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:01.504161 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:01.504413 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:01.504602 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:01.511759 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:01.511891 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:01.512066 17340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:54:01.521320 17340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:01.522452 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:01.523738 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:01.536091 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:01.536098 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:01.536283 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:01.537175 17340 server_base.cc:1034] running on GCE node
I20260927 22:54:01.537856 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:01.538712 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:01.539922 17340 hybrid_clock.cc:648] HybridClock initialized: now 1790549641539885 us; error 49 us; skew 500 ppm
I20260927 22:54:01.540231 17340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:01 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:01.543258 17340 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:54:01.544783 17340 webserver.cc:460] Webserver started at http://127.8.204.129:37859/ using document root <none> and password file <none>
I20260927 22:54:01.545423 17340 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:01.545563 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:01.545866 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:01.548712 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "59054b25d69d49f080f7a7bcf83e1796"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "91a1df2aee7b53bcad2b3d302afa1f9a"
server_key_iv: "451c056b5f22cdbc29c42e2dfc7e1b6b"
server_key_version: "encryptionkey@0"
I20260927 22:54:01.549481 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "59054b25d69d49f080f7a7bcf83e1796"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "91a1df2aee7b53bcad2b3d302afa1f9a"
server_key_iv: "451c056b5f22cdbc29c42e2dfc7e1b6b"
server_key_version: "encryptionkey@0"
I20260927 22:54:01.554486 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:54:01.557966 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:01.559218 17340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:01.559412 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "59054b25d69d49f080f7a7bcf83e1796"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "91a1df2aee7b53bcad2b3d302afa1f9a"
server_key_iv: "451c056b5f22cdbc29c42e2dfc7e1b6b"
server_key_version: "encryptionkey@0"
I20260927 22:54:01.559576 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:01.570250 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:01.573874 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:01.574102 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:01.574919 17340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:01.576221 17340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:01.576323 17340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:01.576470 17340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:01.576541 17340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:01.607424 17340 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:43699
I20260927 22:54:01.607501 17467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:43699 every 8 connection(s)
I20260927 22:54:01.608621 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:01.615924 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17340
I20260927 22:54:01.616124 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:01.616746 9010 external_mini_cluster.cc:1428] Setting key bb8bf500c45179968701171a00d035b0
Sep 27 22:54:01 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549641, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:01.628736 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:01.610977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:37985 (local address 127.8.204.190:45041)
0927 22:54:01.611439 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:01.611444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:01.612467 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 22:54:01.614221 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:01.614226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:01.614322 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:01.614439 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:01.615906 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:01.616751 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:01.618770 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:01.619173 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:01.619265 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:01.623173 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:01.623178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:01.623181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:01.623209 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:01.625634 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:01.626301 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:01.626309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:01.626315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:01.626462 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:01.626865 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:01.626872 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:01.626879 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:01.627238 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:01.627395 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:01.628004 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 22:54:01.628232 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":225,"threads_started":1}
I20260927 22:54:01.629371 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:01.611512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:37985)
0927 22:54:01.612330 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:01.612366 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:01.613969 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:01.614659 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:01.614667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:01.615136 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:01.615748 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:01.615764 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:01.616963 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:01.616968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:01.618633 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:01.618647 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:01.619411 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:01.619420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:01.619737 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:01.620420 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:01.620453 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:01.622946 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:01.625765 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:01.625775 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:01.625789 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:01.626182 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:01.626574 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:01.626583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:01.626585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:01.626749 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:01.627375 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:01.627381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:01.627662 (+ 281us) client_negotiation.cc:769] Sending connection context
0927 22:54:01.627873 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 22:54:01.628220 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":308,"threads_started":1}
I20260927 22:54:01.630813 17468 heartbeater.cc:347] Connected to a master server at 127.8.204.190:45041
I20260927 22:54:01.631134 17468 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:01.631939 17468 heartbeater.cc:511] Master 127.8.204.190:45041 requested a full tablet report, sending...
I20260927 22:54:01.633939 17279 ts_manager.cc:194] Registered new tserver with Master: 59054b25d69d49f080f7a7bcf83e1796 (127.8.204.129:43699)
I20260927 22:54:01.635557 17279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:37985
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:54:01.665853 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45041
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:01.982475 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:01.982755 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:01.982939 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:01.990120 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:01.990259 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:01.990417 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:54:01.999765 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:02.000943 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:02.002182 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:02.014683 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:02.014730 17482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:02.014753 17480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:02.015583 17474 server_base.cc:1034] running on GCE node
I20260927 22:54:02.016161 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:02.017046 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:02.018272 17474 hybrid_clock.cc:648] HybridClock initialized: now 1790549642018227 us; error 59 us; skew 500 ppm
I20260927 22:54:02.018574 17474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549642, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:02.021631 17474 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:54:02.022986 17474 webserver.cc:460] Webserver started at http://127.8.204.130:42931/ using document root <none> and password file <none>
I20260927 22:54:02.023664 17474 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:02.023777 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:02.024016 17474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:02.026686 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "2f318ac360a64b75a089ee175fab1473"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "808daf2ce7662512f81e13798d84dfc4"
server_key_iv: "83363a6ac45b99e6abbe538577c17fb4"
server_key_version: "encryptionkey@0"
I20260927 22:54:02.027448 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "2f318ac360a64b75a089ee175fab1473"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "808daf2ce7662512f81e13798d84dfc4"
server_key_iv: "83363a6ac45b99e6abbe538577c17fb4"
server_key_version: "encryptionkey@0"
I20260927 22:54:02.032377 17474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:54:02.035660 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:02.036840 17474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:02.036983 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "2f318ac360a64b75a089ee175fab1473"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "808daf2ce7662512f81e13798d84dfc4"
server_key_iv: "83363a6ac45b99e6abbe538577c17fb4"
server_key_version: "encryptionkey@0"
I20260927 22:54:02.037168 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:02.049540 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:02.053431 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:02.053694 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:02.054575 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:02.055920 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:02.055989 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:02.056135 17474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:02.056196 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:02.092759 17474 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:39981
I20260927 22:54:02.092800 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:39981 every 8 connection(s)
I20260927 22:54:02.093992 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:02.094897 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17474
I20260927 22:54:02.095088 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:02.095660 9010 external_mini_cluster.cc:1428] Setting key aaa78506cd4c0f38d2343953a7aef5ee
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549642, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:02.115722 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.096808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:37631 (local address 127.8.204.190:45041)
0927 22:54:02.097008 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:02.097015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:02.098397 (+ 1382us) server_negotiation.cc:407] Connection header received
0927 22:54:02.100197 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:02.100205 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:02.100303 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:02.100414 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:02.101784 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.102931 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.105049 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.105433 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.105525 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.109457 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:02.109462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:02.109465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:02.109499 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:02.112869 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.113548 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.113558 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.113560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.113686 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.114108 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.114111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.114114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.114343 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:02.114491 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:02.115135 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 22:54:02.115386 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:54:02.116758 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.097425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:37631)
0927 22:54:02.098246 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:02.098292 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:02.099949 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:02.100614 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:02.100622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:02.101090 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:02.101622 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:02.101639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.103072 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.103076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:02.104904 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:02.104922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.105654 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.105664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:02.105992 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.106686 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:02.106718 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:02.109250 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:02.113006 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:02.113013 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:02.113025 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:02.113420 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:02.113794 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:02.113800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:02.113803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:02.113977 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:02.114464 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:02.114471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:02.114799 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 22:54:02.115040 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 22:54:02.115562 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:54:02.118120 17602 heartbeater.cc:347] Connected to a master server at 127.8.204.190:45041
I20260927 22:54:02.118422 17602 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:02.119182 17602 heartbeater.cc:511] Master 127.8.204.190:45041 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
I20260927 22:54:02.120759 17279 ts_manager.cc:194] Registered new tserver with Master: 2f318ac360a64b75a089ee175fab1473 (127.8.204.130:39981)
I20260927 22:54:02.121513 17279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:37631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:54:02.143679 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45041
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:02.453868 17607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:02.454154 17607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:02.454243 17607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:02.461246 17607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:02.461341 17607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:02.461449 17607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:54:02.470765 17607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:02.471927 17607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:02.473131 17607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:02.485628 17612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:02.486330 17613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:02.486351 17615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:02.486704 17607 server_base.cc:1034] running on GCE node
I20260927 22:54:02.487211 17607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:02.488127 17607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:02.489315 17607 hybrid_clock.cc:648] HybridClock initialized: now 1790549642489273 us; error 37 us; skew 500 ppm
I20260927 22:54:02.489615 17607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549642, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:02.492587 17607 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:54:02.493976 17607 webserver.cc:460] Webserver started at http://127.8.204.131:43205/ using document root <none> and password file <none>
I20260927 22:54:02.494664 17607 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:02.494800 17607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:02.495100 17607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:02.497813 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "82c7ed459c404cf197d290929fcbc670"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "f28cb6082bd4ebbf47057b5049b13dbf"
server_key_iv: "8e13ef56abf5e7bffd73005a87bdaa4b"
server_key_version: "encryptionkey@0"
I20260927 22:54:02.498602 17607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "82c7ed459c404cf197d290929fcbc670"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "f28cb6082bd4ebbf47057b5049b13dbf"
server_key_iv: "8e13ef56abf5e7bffd73005a87bdaa4b"
server_key_version: "encryptionkey@0"
I20260927 22:54:02.503422 17607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 22:54:02.506713 17622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:02.507979 17607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:02.508149 17607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "82c7ed459c404cf197d290929fcbc670"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "f28cb6082bd4ebbf47057b5049b13dbf"
server_key_iv: "8e13ef56abf5e7bffd73005a87bdaa4b"
server_key_version: "encryptionkey@0"
I20260927 22:54:02.508339 17607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:02.528334 17607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:02.532409 17607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:02.532677 17607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:02.533530 17607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:02.534796 17607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:02.534891 17607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:02.535014 17607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:02.535092 17607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:02.567974 17607 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:33929
I20260927 22:54:02.568048 17734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:33929 every 8 connection(s)
I20260927 22:54:02.569262 17607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:02.572216 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17607
I20260927 22:54:02.572376 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:02.572819 9010 external_mini_cluster.cc:1428] Setting key d8a69c2201fec1956d2f517a639b1795
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549642, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:02.588100 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.571750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:50181 (local address 127.8.204.190:45041)
0927 22:54:02.571979 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:02.571987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:02.573275 (+ 1288us) server_negotiation.cc:407] Connection header received
0927 22:54:02.574996 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:02.575001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:02.575087 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:02.575190 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:02.576641 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.577340 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.578748 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.579136 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.579248 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.582484 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:02.582489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:02.582492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:02.582519 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:02.585088 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.585680 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.585683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.585685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.585841 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.586262 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.586265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.586267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.586655 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:02.586778 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:02.587642 (+ 864us) server_negotiation.cc:299] Negotiation successful
0927 22:54:02.587851 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:54:02.589119 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.572220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:50181)
0927 22:54:02.573110 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:02.573154 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:02.574762 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:02.575483 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:02.575491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:02.575960 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:02.576497 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:02.576512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.577463 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.577466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:02.578632 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:02.578643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.579482 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.579492 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:02.579719 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.580288 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:02.580312 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:02.582313 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:02.585195 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:02.585201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:02.585211 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:02.585588 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:02.585945 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:02.585949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:02.585952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:02.586152 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:02.586789 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:02.586796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:02.587196 (+ 400us) client_negotiation.cc:769] Sending connection context
0927 22:54:02.587487 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 22:54:02.588006 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":390,"threads_started":1}
I20260927 22:54:02.590406 17735 heartbeater.cc:347] Connected to a master server at 127.8.204.190:45041
I20260927 22:54:02.590705 17735 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:02.591455 17735 heartbeater.cc:511] Master 127.8.204.190:45041 requested a full tablet report, sending...
I20260927 22:54:02.592790 17279 ts_manager.cc:194] Registered new tserver with Master: 82c7ed459c404cf197d290929fcbc670 (127.8.204.131:33929)
I20260927 22:54:02.593572 17279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:50181
I20260927 22:54:02.599972 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.129@KRBTEST.COM" created.
I20260927 22:54:02.638873 17468 heartbeater.cc:503] Master 127.8.204.190:45041 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab.
Entry for principal oryx/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab.
WARNING: no policy specified for oryx/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab.
Entry for principal oryx/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab.
WARNING: no policy specified for oryx/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab.
Entry for principal oryx/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab.
I20260927 22:54:02.726272 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17340
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:54:02.741833 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.727384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.8.204.129:43699)
0927 22:54:02.727945 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:02.727952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:02.727979 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:54:02.728044 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:02.728050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:02.728269 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:02.728402 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:02.729705 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.730792 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.731958 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.732513 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.732607 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.736403 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:02.736413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:02.736426 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:02.736458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:02.739156 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.740001 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.740008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.740010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.740133 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.740422 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.740425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.740427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.740778 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:02.740983 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:02.741247 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 22:54:02.741418 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":278,"threads_started":1}
I20260927 22:54:02.743001 17463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:02.834302 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17340
I20260927 22:54:02.851442 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17474
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:54:02.865837 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.852251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59004 (local address 127.8.204.130:39981)
0927 22:54:02.852846 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:02.852854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:02.852879 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:54:02.852941 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:02.852947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:02.853162 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:02.853290 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:02.854387 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.855521 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.856854 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.857378 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.857472 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.861206 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:02.861216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:02.861229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:02.861257 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:02.863744 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.864292 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.864299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.864301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.864412 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.864687 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.864690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.864692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.864970 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:02.865189 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:02.865396 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:54:02.865569 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":294,"threads_started":1}
I20260927 22:54:02.867053 17597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:02.947634 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17474
I20260927 22:54:02.963258 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17607
Sep 27 22:54:02 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:54:02.980263 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:02.964334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.8.204.131:33929)
0927 22:54:02.964947 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:02.964953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:02.964980 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:54:02.965039 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:02.965045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:02.965257 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:02.965444 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:02.966763 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.967872 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.969113 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:02.969694 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:02.969842 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:02.974422 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:02.974432 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:02.974447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:02.974486 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:02.977698 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.978335 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.978341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.978343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.978492 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:02.978817 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:02.978820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:02.978821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:02.979152 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:02.979459 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:02.979664 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 22:54:02.979860 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":327,"threads_started":1}
I20260927 22:54:02.981527 17730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:03.064661 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17607
I20260927 22:54:03.082110 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17247
I20260927 22:54:03.083247 17309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:03.183082 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17247
I20260927 22:54:03.203207 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45041
--webserver_interface=127.8.204.190
--webserver_port=40873
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:45041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:03.512887 17752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:03.513167 17752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:03.513338 17752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:03.520473 17752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:03.520601 17752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:03.520690 17752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:03.520787 17752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:03.520869 17752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:03.530486 17752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:45041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:45041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=40873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:03.531788 17752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:03.533038 17752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:03.544705 17757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:03.544587 17760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:03.544615 17758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:03.545367 17752 server_base.cc:1034] running on GCE node
I20260927 22:54:03.546077 17752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:03.547394 17752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:03.548588 17752 hybrid_clock.cc:648] HybridClock initialized: now 1790549643548544 us; error 44 us; skew 500 ppm
I20260927 22:54:03.548861 17752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:03 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549643, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:03.551836 17752 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:54:03.553229 17752 webserver.cc:460] Webserver started at http://127.8.204.190:40873/ using document root <none> and password file <none>
I20260927 22:54:03.553858 17752 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:03.553964 17752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:03.559203 17752 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:03.562091 17767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:03.563634 17752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:54:03.563829 17752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "d62672e841184c8e992effaa48f19dce"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "7ae5c98af33042591d208164d7d2bbaf"
server_key_iv: "e08afb904d792b1a6504ba05c6381d54"
server_key_version: "encryptionkey@0"
I20260927 22:54:03.564786 17752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:03.584755 17752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:03.588568 17752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:03.588835 17752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:03.606652 17752 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:45041
I20260927 22:54:03.606674 17818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:45041 every 8 connection(s)
I20260927 22:54:03.608178 17752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:03.613030 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17752
I20260927 22:54:03.614012 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:43699
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:45041
--webserver_port=37859
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:54:03.615664 17819 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:54:03.620589 17819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Bootstrap starting.
I20260927 22:54:03.638777 17819 log.cc:824] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:03.647022 17819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:54:03.647549 17819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Bootstrap complete.
I20260927 22:54:03.656288 17819 raft_consensus.cc:348] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:03.657099 17819 raft_consensus.cc:729] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d62672e841184c8e992effaa48f19dce, State: Initialized, Role: FOLLOWER
I20260927 22:54:03.657706 17819 consensus_queue.cc:260] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:03.657929 17819 raft_consensus.cc:388] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:03.658037 17819 raft_consensus.cc:482] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:03.658236 17819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:54:03.661907 17819 raft_consensus.cc:504] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:03.662387 17819 leader_election.cc:302] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d62672e841184c8e992effaa48f19dce; no voters:
I20260927 22:54:03.663000 17819 leader_election.cc:288] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:54:03.663126 17823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:54:03.664341 17823 raft_consensus.cc:686] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [term 2 LEADER]: Becoming Leader. State: Replica: d62672e841184c8e992effaa48f19dce, State: Running, Role: LEADER
I20260927 22:54:03.664915 17823 consensus_queue.cc:237] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } }
I20260927 22:54:03.665187 17819 sys_catalog.cc:564] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:03.667346 17825 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader d62672e841184c8e992effaa48f19dce. Latest consensus state: current_term: 2 leader_uuid: "d62672e841184c8e992effaa48f19dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } } }
I20260927 22:54:03.667676 17825 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:03.668192 17824 sys_catalog.cc:455] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d62672e841184c8e992effaa48f19dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d62672e841184c8e992effaa48f19dce" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 45041 } } }
I20260927 22:54:03.668462 17824 sys_catalog.cc:458] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:03.676964 17828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:03.683413 17828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:03.685386 17828 catalog_manager.cc:1259] Loaded cluster ID: aeb291d0a07b4680ae168d600fbfaa17
I20260927 22:54:03.685501 17828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:03.690464 17828 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:03.692293 17828 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d62672e841184c8e992effaa48f19dce: Loaded TSK: 0
I20260927 22:54:03.694132 17828 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:54:03.964956 17821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:03.965256 17821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:03.965444 17821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:03.972898 17821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:03.973037 17821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:03.973210 17821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:54:03.982786 17821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.129.keytab
--principal=oryx/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:43699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=37859
--tserver_master_addrs=127.8.204.190:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:03.984151 17821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:03.985502 17821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:03.999531 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:03.999838 17845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:04.000301 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:04.000761 17821 server_base.cc:1034] running on GCE node
I20260927 22:54:04.001453 17821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:04.002597 17821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:04.003830 17821 hybrid_clock.cc:648] HybridClock initialized: now 1790549644003779 us; error 57 us; skew 500 ppm
I20260927 22:54:04.004177 17821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549644, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:04.007893 17821 init.cc:377] Logged in from keytab as oryx/127.8.204.129@KRBTEST.COM (short username oryx)
I20260927 22:54:04.009459 17821 webserver.cc:460] Webserver started at http://127.8.204.129:37859/ using document root <none> and password file <none>
I20260927 22:54:04.010140 17821 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:04.010293 17821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:04.015728 17821 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:04.018405 17855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.019843 17821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:04.020007 17821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "59054b25d69d49f080f7a7bcf83e1796"
format_stamp: "Formatted at 2026-09-27 22:54:01 on dist-test-slave-40vx"
server_key: "91a1df2aee7b53bcad2b3d302afa1f9a"
server_key_iv: "451c056b5f22cdbc29c42e2dfc7e1b6b"
server_key_version: "encryptionkey@0"
I20260927 22:54:04.020938 17821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:04.055387 17821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:04.059727 17821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:04.059993 17821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:04.060896 17821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:04.062139 17821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:04.062263 17821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.062386 17821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:04.062479 17821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.094494 17821 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:43699
I20260927 22:54:04.094583 17967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:43699 every 8 connection(s)
I20260927 22:54:04.095993 17821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:04.097005 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17821
I20260927 22:54:04.097743 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:39981
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:45041
--webserver_port=42931
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:54:04.417369 17972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:04.417671 17972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:04.417829 17972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:04.424808 17972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:04.424945 17972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:04.425139 17972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:54:04.434370 17972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.130.keytab
--principal=oryx/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:39981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=42931
--tserver_master_addrs=127.8.204.190:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:04.435647 17972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:04.436962 17972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:04.454576 17978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:04.454576 17979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:04.455338 17981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:04.456022 17972 server_base.cc:1034] running on GCE node
I20260927 22:54:04.456636 17972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:04.457597 17972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:04.458978 17972 hybrid_clock.cc:648] HybridClock initialized: now 1790549644458896 us; error 88 us; skew 500 ppm
I20260927 22:54:04.459344 17972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549644, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:04.467715 17972 init.cc:377] Logged in from keytab as oryx/127.8.204.130@KRBTEST.COM (short username oryx)
I20260927 22:54:04.469588 17972 webserver.cc:460] Webserver started at http://127.8.204.130:42931/ using document root <none> and password file <none>
I20260927 22:54:04.470288 17972 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:04.470438 17972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:04.475868 17972 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:54:04.478533 17988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.479934 17972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:04.480083 17972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "2f318ac360a64b75a089ee175fab1473"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "808daf2ce7662512f81e13798d84dfc4"
server_key_iv: "83363a6ac45b99e6abbe538577c17fb4"
server_key_version: "encryptionkey@0"
I20260927 22:54:04.480988 17972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:04.503885 17972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:04.507982 17972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:04.508240 17972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:04.509282 17972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:04.510673 17972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:04.510761 17972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.510901 17972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:04.510982 17972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.554561 17972 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:39981
I20260927 22:54:04.554603 18100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:39981 every 8 connection(s)
I20260927 22:54:04.555985 17972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:04.560529 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 17972
I20260927 22:54:04.561581 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:33929
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:45041
--webserver_port=43205
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:54:04.906622 18105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:04.906953 18105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:04.907130 18105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:04.914273 18105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:04.914407 18105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:04.914597 18105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:54:04.924110 18105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/krb5kdc/oryx_127.8.204.131.keytab
--principal=oryx/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:33929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=43205
--tserver_master_addrs=127.8.204.190:45041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:04.925459 18105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:04.926812 18105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:04.943691 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:04.944555 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:04.956792 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:04.957252 18105 server_base.cc:1034] running on GCE node
I20260927 22:54:04.958307 18105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:04.960018 18105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:04.961455 18105 hybrid_clock.cc:648] HybridClock initialized: now 1790549644961369 us; error 79 us; skew 500 ppm
I20260927 22:54:04.962007 18105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:04 dist-test-slave-40vx krb5kdc[17234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549644, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:04.968024 18105 init.cc:377] Logged in from keytab as oryx/127.8.204.131@KRBTEST.COM (short username oryx)
I20260927 22:54:04.970808 18105 webserver.cc:460] Webserver started at http://127.8.204.131:43205/ using document root <none> and password file <none>
I20260927 22:54:04.972134 18105 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:04.972348 18105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:04.982731 18105 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:54:04.987766 18121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:04.990237 18105 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:54:04.990481 18105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "82c7ed459c404cf197d290929fcbc670"
format_stamp: "Formatted at 2026-09-27 22:54:02 on dist-test-slave-40vx"
server_key: "f28cb6082bd4ebbf47057b5049b13dbf"
server_key_iv: "8e13ef56abf5e7bffd73005a87bdaa4b"
server_key_version: "encryptionkey@0"
I20260927 22:54:04.992543 18105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:05.045250 18105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:05.053283 18105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:05.053714 18105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:05.055562 18105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:05.058085 18105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:05.058244 18105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:05.058393 18105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:05.058501 18105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:05.139138 18105 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:33929
I20260927 22:54:05.139421 18233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:33929 every 8 connection(s)
I20260927 22:54:05.141794 18105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:05.145506 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18105
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:05.189945 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:05.146730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.8.204.190:45041)
0927 22:54:05.159828 (+ 13098us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:05.159838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:05.159866 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:54:05.159952 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:05.159961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:05.160082 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:05.163112 (+ 3030us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:05.163490 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.166442 (+ 2952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.169202 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.170192 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.170376 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.175664 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:05.175682 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:05.175717 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:05.175800 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:05.182189 (+ 6389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:05.183515 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:05.183529 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:05.183544 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:05.183840 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:05.184507 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:05.184514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:05.184520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:05.185403 (+ 883us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:05.185814 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:05.187729 (+ 1915us) server_negotiation.cc:299] Negotiation successful
0927 22:54:05.188424 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12929,"thread_start_us":17713,"threads_started":1}
W20260927 22:54:05.557751 17971 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.442s user 0.000s sys 0.016s
W20260927 22:54:05.559800 17970 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:04.099297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:47825 (local address 127.8.204.190:45041)
0927 22:54:04.100044 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:04.100060 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:04.103983 (+ 3923us) server_negotiation.cc:407] Connection header received
0927 22:54:04.107116 (+ 3133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:04.107129 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:04.107510 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:04.107819 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:04.109791 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.110986 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.113604 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.114252 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.114383 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.559292 (+1444909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:47825: BlockingRecv error: recv got EOF from 127.8.204.129:47825 (error 108)
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":278,"threads_started":1}
I20260927 22:54:05.560380 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:04.099357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:47825)
0927 22:54:04.103780 (+ 4423us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:04.103828 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:04.106581 (+ 2753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:04.108009 (+ 1428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:04.108021 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:04.108737 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:04.109477 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:04.109501 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.111155 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.111163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:04.113439 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:04.113458 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.114484 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.114493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:04.114837 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:04.115623 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:04.115661 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:05.558877 (+1443216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4107,"thread_start_us":3932,"threads_started":1}
W20260927 22:54:05.561502 17968 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:45041 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.204.190:45041: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:54:05.582721 17970 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:05.562823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:37989 (local address 127.8.204.190:45041)
0927 22:54:05.563083 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:05.563093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:05.563801 (+ 708us) server_negotiation.cc:407] Connection header received
0927 22:54:05.564179 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:05.564187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:05.564299 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:05.564396 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:05.566145 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.568014 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.571499 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.572484 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.572615 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.582351 (+ 9736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:37989: BlockingRecv error: recv got EOF from 127.8.204.129:37989 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:54:05.583817 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:05.563339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:37989)
0927 22:54:05.563681 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:05.563709 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:05.563938 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:05.564691 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:05.564699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:05.565530 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:05.565828 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.565844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.568317 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.568326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.570896 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.570918 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.573223 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.573232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.573501 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.575531 (+ 2030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:05.575569 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:05.583392 (+ 7823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:05.601187 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:05.585117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:56077)
0927 22:54:05.585565 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:05.585593 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:05.585740 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:05.586779 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:05.586786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:05.587346 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:05.587707 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.587724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.589873 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.589881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.592231 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.592251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.593657 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.593666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.593972 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.594835 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:05.594869 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:05.600753 (+ 5884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269}
W20260927 22:54:05.601742 17968 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:54:05.602860 17970 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:05.586125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:56077 (local address 127.8.204.190:45041)
0927 22:54:05.586333 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:05.586342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:05.586368 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:54:05.586452 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:05.586460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:05.586558 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:05.586694 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:05.587955 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.589615 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.592430 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.593445 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.593630 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.602130 (+ 8500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:56077: BlockingRecv error: recv got EOF from 127.8.204.129:56077 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:54:05.865546 18104 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.297s user 0.000s sys 0.015s
W20260927 22:54:05.867187 18103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:04.559396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:50781 (local address 127.8.204.190:45041)
0927 22:54:04.559930 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:04.559936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:04.560807 (+ 871us) server_negotiation.cc:407] Connection header received
0927 22:54:04.562809 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:04.562814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:04.562895 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:04.563023 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:04.564683 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.565586 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.567239 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.567675 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.567810 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.866953 (+1299143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:50781: BlockingRecv error: recv got EOF from 127.8.204.130:50781 (error 108)
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":282,"threads_started":1}
I20260927 22:54:05.868546 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:04.559830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:50781)
0927 22:54:04.560666 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:04.560709 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:04.562543 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:04.563256 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:04.563265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:04.563950 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:04.564512 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:04.564528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.565730 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.565738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:04.567098 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:04.567109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:04.567903 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:04.567913 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:04.568149 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:04.568892 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:04.568918 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:05.866820 (+1297902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":411,"threads_started":1}
W20260927 22:54:05.869616 18101 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:45041 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.204.190:45041: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:54:05.884791 18103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:05.870316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:48931 (local address 127.8.204.190:45041)
0927 22:54:05.870478 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:05.870483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:05.870653 (+ 170us) server_negotiation.cc:407] Connection header received
0927 22:54:05.870871 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:05.870874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:05.870933 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:05.871018 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:05.872419 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.874167 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.876991 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.877382 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.877478 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.884564 (+ 7086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:48931: BlockingRecv error: recv got EOF from 127.8.204.130:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:54:05.884871 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:05.870268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:48931)
0927 22:54:05.870532 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:05.870562 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:05.870743 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:05.871134 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:05.871142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:05.871856 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:05.872162 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.872189 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.874461 (+ 2272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.874483 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.876813 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.876835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.877610 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.877620 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.877879 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.878633 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:05.878668 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:05.884399 (+ 5731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:05 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:05.897722 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:05.885642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:43163)
0927 22:54:05.885939 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:05.885964 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:05.886167 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:05.886859 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:05.886867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:05.887443 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:05.887737 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.887753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.888975 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.888984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.891231 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.891251 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.891990 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.891999 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.892284 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.892989 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:05.893024 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:05.897442 (+ 4418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152}
W20260927 22:54:05.897960 18103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:05.885827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:43163 (local address 127.8.204.190:45041)
0927 22:54:05.885976 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:05.885980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:05.886047 (+ 67us) server_negotiation.cc:407] Connection header received
0927 22:54:05.886351 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:05.886359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:05.886491 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:05.886588 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:05.887913 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.888816 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.891436 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.891823 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.891920 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.897628 (+ 5708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:43163: BlockingRecv error: recv got EOF from 127.8.204.130:43163 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 22:54:05.898274 18101 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:54:06.371519 18238 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.193s user 0.001s sys 0.025s
W20260927 22:54:06.372592 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:05.154165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:58065 (local address 127.8.204.190:45041)
0927 22:54:05.163753 (+ 9588us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:05.163763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:05.163803 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:54:05.163886 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:05.163894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:05.164012 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:05.164185 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:05.168416 (+ 4231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.171930 (+ 3514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.175795 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.176871 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.178357 (+ 1486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.372373 (+1194016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:58065: BlockingRecv error: recv got EOF from 127.8.204.131:58065 (error 108)
Metrics: {"server-negotiator.queue_time_us":9356,"thread_start_us":5363,"threads_started":1}
I20260927 22:54:06.373111 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:05.153568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:58065)
0927 22:54:05.154793 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:05.154857 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:05.158514 (+ 3657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:05.164587 (+ 6073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:05.164607 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:05.165828 (+ 1221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:05.168084 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.168124 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.172195 (+ 4071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.172205 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.175446 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:05.175472 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:05.177255 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:05.177267 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:05.177737 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:05.178736 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:05.178778 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:06.372183 (+1193405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":612,"threads_started":1}
W20260927 22:54:06.373919 18234 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:45041 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.204.190:45041: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:06.382185 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:06.374590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:34483)
0927 22:54:06.374784 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:06.374811 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:06.374922 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:06.375242 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:06.375248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:06.375655 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:06.375808 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.375816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.376708 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.376711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.377612 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.377621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.378227 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.378231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.378331 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.378825 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:06.378839 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:06.382000 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:54:06.382369 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:06.374641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:34483 (local address 127.8.204.190:45041)
0927 22:54:06.374869 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:06.374878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:06.374898 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:54:06.375036 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:06.375039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:06.375099 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:06.375175 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:06.375920 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.376599 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.377719 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.378095 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.378182 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.382172 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:34483: BlockingRecv error: recv got EOF from 127.8.204.131:34483 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:06.390590 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:06.382835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:45527)
0927 22:54:06.382972 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:06.382995 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:06.383062 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:06.383415 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:06.383418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:06.383677 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:06.383805 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.383811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.384746 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.384750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.385736 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.385746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.386688 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.386692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.386891 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.387529 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:06.387550 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:06.390399 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:54:06.390789 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:06.382924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:45527 (local address 127.8.204.190:45041)
0927 22:54:06.383128 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:06.383132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:06.383146 (+ 14us) server_negotiation.cc:407] Connection header received
0927 22:54:06.383180 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:06.383183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:06.383238 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:06.383403 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:06.383908 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.384642 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.385879 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.386550 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.386674 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.390568 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:45527: BlockingRecv error: recv got EOF from 127.8.204.131:45527 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260927 22:54:06.390903 18234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:06.611492 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:06.602220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:50625)
0927 22:54:06.602711 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:06.602745 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:06.602866 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:06.603161 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:06.603164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:06.603585 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:06.603731 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.603739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.604644 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.604647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.605772 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.605783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.606511 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.606517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.606691 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.607348 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:06.607370 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:06.611142 (+ 3772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":301,"threads_started":1}
W20260927 22:54:06.611606 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:06.602302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:50625 (local address 127.8.204.190:45041)
0927 22:54:06.602470 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:06.602474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:06.602830 (+ 356us) server_negotiation.cc:407] Connection header received
0927 22:54:06.602959 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:06.602962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:06.603032 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:06.603116 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:06.603846 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.604508 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.605883 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.606309 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.606460 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.611365 (+ 4905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:50625: BlockingRecv error: recv got EOF from 127.8.204.129:50625 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:06 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:06.907608 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:06.898753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:43143)
0927 22:54:06.899337 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:06.899354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:06.899461 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:06.899736 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:06.899739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:06.900117 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:06.900257 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.900265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.901262 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.901267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.902217 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:06.902229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.903178 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.903184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:06.903403 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.904052 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:06.904069 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:06.907384 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":320,"threads_started":1}
W20260927 22:54:06.907801 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:06.898820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:43143 (local address 127.8.204.190:45041)
0927 22:54:06.898971 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:06.898975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:06.899424 (+ 449us) server_negotiation.cc:407] Connection header received
0927 22:54:06.899556 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:06.899559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:06.899618 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:06.899697 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:06.900413 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.901125 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.902372 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:06.902981 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:06.903122 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:06.907578 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:43143: BlockingRecv error: recv got EOF from 127.8.204.130:43143 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 22:54:07 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:07 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:07.400785 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:07.391633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:50729)
0927 22:54:07.392195 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:07.392211 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:07.392320 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:07.392627 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:07.392630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:07.392988 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:07.393121 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:07.393128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.394212 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.394217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:07.395576 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:07.395587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.396535 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.396541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:07.396712 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:07.397292 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:07.397307 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:07.400515 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":320,"threads_started":1}
W20260927 22:54:07.400941 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:07.391712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:50729 (local address 127.8.204.190:45041)
0927 22:54:07.391865 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:07.391870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:07.392360 (+ 490us) server_negotiation.cc:407] Connection header received
0927 22:54:07.392412 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:07.392415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:07.392487 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:07.392591 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:07.393233 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.394040 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.395706 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.396363 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.396480 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:07.400740 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:50729: BlockingRecv error: recv got EOF from 127.8.204.131:50729 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 22:54:07 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:07 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:07.621371 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:07.612734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:52113)
0927 22:54:07.613221 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:07.613239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:07.613366 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:07.613681 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:07.613686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:07.614044 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:07.614205 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:07.614213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.615158 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.615161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:07.616061 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:07.616071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.616674 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.616678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:07.616789 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:07.617393 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:07.617409 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:07.621133 (+ 3724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
W20260927 22:54:07.621537 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:07.612782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:52113 (local address 127.8.204.190:45041)
0927 22:54:07.613017 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:07.613023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:07.613319 (+ 296us) server_negotiation.cc:407] Connection header received
0927 22:54:07.613467 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:07.613470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:07.613526 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:07.613620 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:07.614316 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.615034 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.616176 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.616554 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.616641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:07.621315 (+ 4674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:52113: BlockingRecv error: recv got EOF from 127.8.204.129:52113 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:54:07 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:07 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:07.917743 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:07.908477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:53047)
0927 22:54:07.908966 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:07.908982 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:07.909102 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:07.909391 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:07.909394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:07.909816 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:07.910134 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:07.910146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.911494 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.911498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:07.912380 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:07.912390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.913291 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.913297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:07.913434 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:07.914131 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:07.914172 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:07.917514 (+ 3342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":293,"threads_started":1}
W20260927 22:54:07.917897 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:07.908602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:53047 (local address 127.8.204.190:45041)
0927 22:54:07.908779 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:07.908784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:07.909067 (+ 283us) server_negotiation.cc:407] Connection header received
0927 22:54:07.909200 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:07.909204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:07.909268 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:07.909354 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:07.910345 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.911375 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.912537 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:07.913115 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:07.913262 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:07.917689 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:53047: BlockingRecv error: recv got EOF from 127.8.204.130:53047 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 22:54:08 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:08 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:08.410409 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:08.401815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:55217)
0927 22:54:08.402315 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:08.402330 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:08.402438 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:08.402724 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:08.402727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:08.403265 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:08.403512 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:08.403534 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.404677 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.404684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:08.405596 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:08.405607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.406288 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.406294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:08.406468 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:08.406996 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:08.407011 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:08.410176 (+ 3165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":285,"threads_started":1}
W20260927 22:54:08.410579 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:08.402069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:55217 (local address 127.8.204.190:45041)
0927 22:54:08.402236 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:08.402241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:08.402402 (+ 161us) server_negotiation.cc:407] Connection header received
0927 22:54:08.402531 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:08.402535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:08.402604 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:08.402684 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:08.403668 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.404538 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.405735 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.406131 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.406257 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:08.410344 (+ 4087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:55217: BlockingRecv error: recv got EOF from 127.8.204.131:55217 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:54:08 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:08 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:08.631047 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:08.622295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:35981)
0927 22:54:08.622770 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:08.622785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:08.622940 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:08.623261 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:08.623266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:08.623667 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:08.623822 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:08.623830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.624790 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.624798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:08.625741 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:08.625752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.626451 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.626456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:08.626642 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:08.627164 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:08.627179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:08.630689 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
W20260927 22:54:08.631140 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:08.622382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:35981 (local address 127.8.204.190:45041)
0927 22:54:08.622637 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:08.622643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:08.622875 (+ 232us) server_negotiation.cc:407] Connection header received
0927 22:54:08.623076 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:08.623080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:08.623135 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:08.623236 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:08.623954 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.624664 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.625874 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.626299 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.626443 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:08.630907 (+ 4464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:35981: BlockingRecv error: recv got EOF from 127.8.204.129:35981 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 22:54:08 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:08 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:08.927202 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:08.918611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:48123)
0927 22:54:08.919127 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:08.919142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:08.919251 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:08.919568 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:08.919571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:08.919892 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:08.920040 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:08.920047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.920984 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.920987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:08.921877 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:08.921888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.922689 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.922693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:08.922892 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:08.923548 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:08.923573 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:08.926977 (+ 3404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
W20260927 22:54:08.927400 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:08.918704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:48123 (local address 127.8.204.190:45041)
0927 22:54:08.918931 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:08.918936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:08.919247 (+ 311us) server_negotiation.cc:407] Connection header received
0927 22:54:08.919395 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:08.919398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:08.919457 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:08.919543 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:08.920166 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.920864 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.921991 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:08.922500 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:08.922666 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:08.927153 (+ 4487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:48123: BlockingRecv error: recv got EOF from 127.8.204.130:48123 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 27 22:54:09 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:09 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:09.419116 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:09.411248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:47349)
0927 22:54:09.411754 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:09.411771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:09.411877 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:09.412158 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:09.412161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:09.412484 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:09.412640 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:09.412648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.413546 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.413549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:09.414423 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:09.414432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.415039 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.415043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:09.415164 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:09.415761 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:09.415777 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:09.418893 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":251,"threads_started":1}
W20260927 22:54:09.419310 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:09.411387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:47349 (local address 127.8.204.190:45041)
0927 22:54:09.411605 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:09.411613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:09.411841 (+ 228us) server_negotiation.cc:407] Connection header received
0927 22:54:09.411965 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:09.411969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:09.412040 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:09.412118 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:09.412745 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.413425 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.414538 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.414917 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.415004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:09.419065 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:47349: BlockingRecv error: recv got EOF from 127.8.204.131:47349 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 22:54:09 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:09 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:09.643622 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:09.632095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:46677)
0927 22:54:09.632588 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:09.632604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:09.632757 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:09.633097 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:09.633101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:09.633472 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:09.633635 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:09.633643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.634609 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.634617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:09.635598 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:09.635608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.636250 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.636255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:09.636432 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:09.640125 (+ 3693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:09.640142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:09.643230 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":275,"threads_started":1}
W20260927 22:54:09.643733 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:09.632125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:46677 (local address 127.8.204.190:45041)
0927 22:54:09.632321 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:09.632327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:09.632695 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:54:09.632889 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:09.632894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:09.632961 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:09.633067 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:09.633774 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.634486 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.635733 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.636112 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.636241 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:09.643456 (+ 7215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:46677: BlockingRecv error: recv got EOF from 127.8.204.129:46677 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 22:54:09 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:09 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:09.937093 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:09.928327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:35123)
0927 22:54:09.928876 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:09.928892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:09.929045 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:09.929420 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:09.929425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:09.929937 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:09.930156 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:09.930165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.931182 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.931185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:09.932217 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:09.932227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.932856 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.932860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:09.932969 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:09.933594 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:09.933611 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:09.936862 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":313,"threads_started":1}
W20260927 22:54:09.937243 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:09.928422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:35123 (local address 127.8.204.190:45041)
0927 22:54:09.928690 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:09.928697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:09.928981 (+ 284us) server_negotiation.cc:407] Connection header received
0927 22:54:09.929189 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:09.929195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:09.929277 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:09.929383 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:09.930321 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.931071 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.932333 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:09.932743 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:09.932831 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:09.937032 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:35123: BlockingRecv error: recv got EOF from 127.8.204.130:35123 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 27 22:54:10 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:10 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:10.429337 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:10.420079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:35193)
0927 22:54:10.420589 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:10.420604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:10.420758 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:10.421124 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:10.421129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:10.421573 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:10.421737 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:10.421746 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.423003 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.423009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:10.424270 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:10.424282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.425165 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.425171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:10.425388 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.425999 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:10.426015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:10.429086 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":279,"threads_started":1}
W20260927 22:54:10.429553 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:10.420154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:35193 (local address 127.8.204.190:45041)
0927 22:54:10.420401 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:10.420408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:10.420693 (+ 285us) server_negotiation.cc:407] Connection header received
0927 22:54:10.420889 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:10.420895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:10.420996 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:10.421099 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:10.422102 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.422833 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.424427 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.424967 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.425126 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.429273 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:35193: BlockingRecv error: recv got EOF from 127.8.204.131:35193 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 22:54:10 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:10 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:10.653393 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:10.644758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:59693)
0927 22:54:10.645240 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:10.645257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:10.645366 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:10.645655 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:10.645659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:10.646065 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:10.646209 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:10.646217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.647142 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.647146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:10.648188 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:10.648201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.648957 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.648962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:10.649160 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.649889 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:10.649910 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:10.653180 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":273,"threads_started":1}
W20260927 22:54:10.653610 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:10.644956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:59693 (local address 127.8.204.190:45041)
0927 22:54:10.645125 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:10.645129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:10.645328 (+ 199us) server_negotiation.cc:407] Connection header received
0927 22:54:10.645460 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:10.645463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:10.645524 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:10.645604 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:10.646324 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.647027 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.648321 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.648812 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.648944 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.653393 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:59693: BlockingRecv error: recv got EOF from 127.8.204.129:59693 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 22:54:10 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:10 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:10.946946 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:10.937914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:51005)
0927 22:54:10.938453 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:10.938470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:10.938637 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:10.938967 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:10.938971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:10.939434 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:10.939585 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:10.939594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.940649 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.940656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:10.941589 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:10.941600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.942319 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.942325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:10.942520 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.943182 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:10.943200 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:10.946607 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":302,"threads_started":1}
W20260927 22:54:10.947072 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:10.938120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:51005 (local address 127.8.204.190:45041)
0927 22:54:10.938326 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:10.938334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:10.938573 (+ 239us) server_negotiation.cc:407] Connection header received
0927 22:54:10.938768 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:10.938775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:10.938838 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:10.938938 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:10.939728 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.940506 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.941732 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:10.942168 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:10.942296 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:10.946815 (+ 4519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:51005: BlockingRecv error: recv got EOF from 127.8.204.130:51005 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:54:11 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:11 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:11.439404 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:11.430292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:51445)
0927 22:54:11.430877 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:11.430894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:11.431051 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:11.431435 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:11.431440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:11.431920 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:11.432060 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.432069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.433212 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.433219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.434160 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.434171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.434870 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.434875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.435076 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.435763 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:11.435780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:11.439042 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":293,"threads_started":1}
W20260927 22:54:11.439469 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:11.430347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:51445 (local address 127.8.204.190:45041)
0927 22:54:11.430615 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:11.430624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:11.430984 (+ 360us) server_negotiation.cc:407] Connection header received
0927 22:54:11.431201 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:11.431205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:11.431294 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:11.431408 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:11.432206 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.433087 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.434300 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.434714 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.434864 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.439204 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:51445: BlockingRecv error: recv got EOF from 127.8.204.131:51445 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 22:54:11 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:11 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:11.663156 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:11.654310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:60293)
0927 22:54:11.654828 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:11.654844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:11.654970 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:11.655256 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:11.655259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:11.655705 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:11.655849 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.655857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.656866 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.656871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.657806 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.657817 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.658462 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.658466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.658595 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.659318 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:11.659339 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:11.662942 (+ 3603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":297,"threads_started":1}
W20260927 22:54:11.663472 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:11.654395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:60293 (local address 127.8.204.190:45041)
0927 22:54:11.654580 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:11.654585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:11.654963 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:54:11.655071 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:11.655074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:11.655136 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:11.655214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:11.656003 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.656735 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.657923 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.658327 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.658414 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.663143 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:60293: BlockingRecv error: recv got EOF from 127.8.204.129:60293 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:54:11 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:11 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:11.956194 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:11.948059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:55875)
0927 22:54:11.948539 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:11.948555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:11.948662 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:11.948958 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:11.948960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:11.949330 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:11.949491 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.949499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.950414 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.950418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.951388 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:11.951399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.952054 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.952058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:11.952180 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.952695 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:11.952711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:11.955974 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":261,"threads_started":1}
W20260927 22:54:11.956398 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:11.948134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:55875 (local address 127.8.204.190:45041)
0927 22:54:11.948284 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:11.948288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:11.948636 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:54:11.948749 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:11.948754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:11.948829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:11.948928 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:11.949626 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.950296 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.951501 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:11.951923 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:11.952015 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:11.956166 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:55875: BlockingRecv error: recv got EOF from 127.8.204.130:55875 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 22:54:12 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:12 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:12.448611 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:12.440401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:45525)
0927 22:54:12.440893 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:12.440908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:12.441022 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:12.441293 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:12.441296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:12.441647 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:12.441801 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.441808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.442762 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.442765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.443778 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.443789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.444475 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.444481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.444670 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.445192 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:12.445208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:12.448379 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
W20260927 22:54:12.448760 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:12.440465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:45525 (local address 127.8.204.190:45041)
0927 22:54:12.440638 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:12.440642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:12.440977 (+ 335us) server_negotiation.cc:407] Connection header received
0927 22:54:12.441117 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:12.441120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:12.441175 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:12.441252 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:12.441936 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.442658 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.443915 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.444322 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.444457 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.448548 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:45525: BlockingRecv error: recv got EOF from 127.8.204.131:45525 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 22:54:12 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:12 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:12.673255 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:12.664059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:51277)
0927 22:54:12.664643 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:12.664659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:12.664825 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:12.665181 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:12.665186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:12.665633 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:12.665824 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.665833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.666833 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.666840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.667794 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.667805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.668572 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.668577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.668768 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.669476 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:12.669493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:12.673033 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":247,"threads_started":1}
W20260927 22:54:12.673442 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:12.664174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:51277 (local address 127.8.204.190:45041)
0927 22:54:12.664400 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:12.664408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:12.664760 (+ 352us) server_negotiation.cc:407] Connection header received
0927 22:54:12.664974 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:12.664979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:12.665050 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:12.665152 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:12.665966 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.666704 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.667934 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.668411 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.668536 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.673222 (+ 4686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:51277: BlockingRecv error: recv got EOF from 127.8.204.129:51277 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:54:12 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:12 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:12.966063 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:12.957044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:46579)
0927 22:54:12.957532 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:12.957548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:12.957708 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:12.958050 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:12.958056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:12.958471 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:12.958633 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.958641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.959730 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.959736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.960666 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:12.960677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.961397 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.961403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:12.961566 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.962139 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:12.962155 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:12.965718 (+ 3563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":220,"threads_started":1}
W20260927 22:54:12.966145 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:12.957111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:46579 (local address 127.8.204.190:45041)
0927 22:54:12.957307 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:12.957315 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:12.957637 (+ 322us) server_negotiation.cc:407] Connection header received
0927 22:54:12.957845 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:12.957850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:12.957917 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:12.958014 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:12.958774 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.959599 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.960806 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:12.961248 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:12.961382 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:12.965942 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:46579: BlockingRecv error: recv got EOF from 127.8.204.130:46579 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 22:54:13 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:13 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:13.458144 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.449474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:53407)
0927 22:54:13.450053 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:13.450069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:13.450180 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:13.450465 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:13.450468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:13.450889 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:13.451047 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.451055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.452071 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.452075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.453057 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.453069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.453688 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.453693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.453802 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.454391 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:13.454407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:13.457893 (+ 3486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":285,"threads_started":1}
W20260927 22:54:13.458309 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:13.449593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:53407 (local address 127.8.204.190:45041)
0927 22:54:13.449865 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.449871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.450148 (+ 277us) server_negotiation.cc:407] Connection header received
0927 22:54:13.450283 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.450287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.450343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.450419 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.451164 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.451928 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.453178 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.453555 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.453642 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.458078 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:53407: BlockingRecv error: recv got EOF from 127.8.204.131:53407 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 27 22:54:13 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:13 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:13.682780 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.674247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:33999)
0927 22:54:13.674800 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:13.674820 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:13.674956 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:13.675248 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:13.675251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:13.675696 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:13.675865 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.675873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.676823 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.676826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.677722 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.677731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.678390 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.678394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.678498 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.679048 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:13.679064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:13.682517 (+ 3453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":271,"threads_started":1}
W20260927 22:54:13.682873 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:13.674384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:33999 (local address 127.8.204.190:45041)
0927 22:54:13.674622 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.674630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.674910 (+ 280us) server_negotiation.cc:407] Connection header received
0927 22:54:13.675066 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.675069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.675128 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.675208 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.675986 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.676710 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.677835 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.678242 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.678335 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.682692 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:33999: BlockingRecv error: recv got EOF from 127.8.204.129:33999 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:54:13 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:13 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:13.976405 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:13.967066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:60925)
0927 22:54:13.967640 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:13.967656 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:13.967807 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:13.968161 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:13.968165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:13.968582 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:13.968758 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.968783 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.969971 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.969977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.971373 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:13.971385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.972300 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.972305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:13.972423 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.972980 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:13.973007 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:13.976171 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
W20260927 22:54:13.976577 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:13.967154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:60925 (local address 127.8.204.190:45041)
0927 22:54:13.967399 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:13.967408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:13.967735 (+ 327us) server_negotiation.cc:407] Connection header received
0927 22:54:13.967945 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:13.967949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:13.968009 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:13.968113 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:13.968908 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.969830 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.971551 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:13.972143 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:13.972238 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:13.976370 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:60925: BlockingRecv error: recv got EOF from 127.8.204.130:60925 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 22:54:14 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:14 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:14.468061 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:14.459165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:60983)
0927 22:54:14.459781 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:14.459797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:14.459906 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:14.460184 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:14.460187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:14.460573 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:14.460740 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.460748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.461721 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.461724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.462642 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.462653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.463376 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.463381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.463527 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.464088 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:14.464104 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:14.467763 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":354,"threads_started":1}
W20260927 22:54:14.468245 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:14.459206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:60983 (local address 127.8.204.190:45041)
0927 22:54:14.459408 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:14.459413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:14.459870 (+ 457us) server_negotiation.cc:407] Connection header received
0927 22:54:14.459999 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:14.460003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:14.460061 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:14.460139 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:14.460866 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.461609 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.462759 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.463166 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.463255 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.468004 (+ 4749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:60983: BlockingRecv error: recv got EOF from 127.8.204.131:60983 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 22:54:14 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:14 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:14.704398 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:14.684025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:49657)
0927 22:54:14.691671 (+ 7646us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:14.691689 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:14.691853 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:14.692194 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:14.692198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:14.692629 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:14.692773 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.692782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.693761 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.693766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.694714 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.694725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.695440 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.695445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.695693 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.700609 (+ 4916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:14.700640 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:14.704072 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7528,"thread_start_us":7397,"threads_started":1}
W20260927 22:54:14.704512 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:14.684171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:49657 (local address 127.8.204.190:45041)
0927 22:54:14.684363 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:14.684368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:14.691779 (+ 7411us) server_negotiation.cc:407] Connection header received
0927 22:54:14.691973 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:14.691981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:14.692061 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:14.692159 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:14.692910 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.693626 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.694850 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.695263 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.695426 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.704264 (+ 8838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:49657: BlockingRecv error: recv got EOF from 127.8.204.129:49657 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 22:54:14 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:14 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:14.986610 18266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:14.977389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:50911)
0927 22:54:14.977964 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:14.977980 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:14.978161 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:14.978558 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:14.978566 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:14.979069 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:14.979239 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.979249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.980255 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.980259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.981309 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:14.981320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.981956 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.981960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:14.982099 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.982703 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:14.982722 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:14.986329 (+ 3607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":341,"threads_started":1}
W20260927 22:54:14.986779 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:14.977449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:50911 (local address 127.8.204.190:45041)
0927 22:54:14.977646 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:14.977655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:14.978074 (+ 419us) server_negotiation.cc:407] Connection header received
0927 22:54:14.978287 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:14.978292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:14.978364 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:14.978478 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:14.979428 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.980142 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.981432 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:14.981819 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:14.981916 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:14.986542 (+ 4626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:50911: BlockingRecv error: recv got EOF from 127.8.204.130:50911 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:54:15 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:15 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:15.477697 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:15.468993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:54975)
0927 22:54:15.469462 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:15.469478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:15.469641 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:15.469971 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:15.469976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:15.470395 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:15.470539 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.470549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.471603 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.471611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.472557 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.472567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.473271 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.473277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.473446 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.474010 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:15.474025 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:15.477470 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
W20260927 22:54:15.477852 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:15.469013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:54975 (local address 127.8.204.190:45041)
0927 22:54:15.469222 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:15.469230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:15.469578 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:54:15.469756 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:15.469761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:15.469847 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:15.469944 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:15.470681 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.471484 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.472695 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.473123 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.473260 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.477652 (+ 4392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:54975: BlockingRecv error: recv got EOF from 127.8.204.131:54975 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 22:54:15 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:15 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:15.714548 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:15.705638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:58975)
0927 22:54:15.706196 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:15.706213 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:15.706391 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:15.706724 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:15.706729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:15.707110 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:15.707259 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.707325 (+ 66us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.708412 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.708419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.709394 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.709406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.710173 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.710178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.710357 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.710926 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:15.710943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:15.714175 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":307,"threads_started":1}
W20260927 22:54:15.714632 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:15.705800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:58975 (local address 127.8.204.190:45041)
0927 22:54:15.706061 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:15.706067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:15.706304 (+ 237us) server_negotiation.cc:407] Connection header received
0927 22:54:15.706519 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:15.706526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:15.706588 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:15.706691 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:15.707468 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.708259 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.709536 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.710007 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.710159 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.714356 (+ 4197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:58975: BlockingRecv error: recv got EOF from 127.8.204.129:58975 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 22:54:15 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:15 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:15.997064 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:15.987504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:51205)
0927 22:54:15.988052 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:15.988068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:15.988181 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:15.988447 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:15.988450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:15.988819 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:15.988978 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.988987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.989910 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.989913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.990852 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:15.990863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.991497 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.991502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:15.991621 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.992265 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:15.992282 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:15.996756 (+ 4474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
W20260927 22:54:15.997164 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:15.987530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:51205 (local address 127.8.204.190:45041)
0927 22:54:15.987678 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:15.987683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:15.988138 (+ 455us) server_negotiation.cc:407] Connection header received
0927 22:54:15.988270 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:15.988274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:15.988331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:15.988410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:15.989109 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.989793 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.990970 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:15.991353 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:15.991441 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:15.996932 (+ 5491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:51205: BlockingRecv error: recv got EOF from 127.8.204.130:51205 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 22:54:16 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:16 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:16.487875 18270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.478567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:38137)
0927 22:54:16.479120 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.479136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.479321 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.479671 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.479676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.480072 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:16.480225 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.480234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.481248 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.481256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.482230 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.482241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.483001 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.483007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.483174 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.483808 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:16.483824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:16.487533 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":326,"threads_started":1}
W20260927 22:54:16.487988 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:16.478736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38137 (local address 127.8.204.190:45041)
0927 22:54:16.479014 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.479022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.479223 (+ 201us) server_negotiation.cc:407] Connection header received
0927 22:54:16.479464 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.479470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.479534 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.479639 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.480377 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.481114 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.482395 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.482838 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.482978 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.487721 (+ 4743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:38137: BlockingRecv error: recv got EOF from 127.8.204.131:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 27 22:54:16 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:16 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:16.724189 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.715475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:43313)
0927 22:54:16.715914 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.715929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.716088 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.716423 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.716427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.716850 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:16.717022 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.717031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.717997 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.718003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.718916 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.718926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.719710 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.719716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:16.719898 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.720456 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:16.720472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:16.723877 (+ 3405us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
W20260927 22:54:16.724370 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:16.715522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:43313 (local address 127.8.204.190:45041)
0927 22:54:16.715734 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.715742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.716021 (+ 279us) server_negotiation.cc:407] Connection header received
0927 22:54:16.716209 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.716214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.716290 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.716389 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.717164 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.717864 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.719065 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:16.719566 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:16.719697 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:16.724124 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:43313: BlockingRecv error: recv got EOF from 127.8.204.129:43313 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 22:54:17 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:17 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:17.006913 18272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:16.997929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:54733)
0927 22:54:16.998430 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:16.998446 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:16.998563 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:16.998859 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:16.998862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:16.999386 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:16.999646 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:16.999662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.001068 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.001072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:17.002096 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:17.002107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.002708 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.002712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:17.002824 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.003484 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:17.003502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:17.006684 (+ 3182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":286,"threads_started":1}
W20260927 22:54:17.007118 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:16.998048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:54733 (local address 127.8.204.190:45041)
0927 22:54:16.998209 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:16.998213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:16.998524 (+ 311us) server_negotiation.cc:407] Connection header received
0927 22:54:16.998656 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:16.998660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:16.998724 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:16.998805 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:16.999812 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.000952 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.002208 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.002594 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.002684 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.006875 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:54733: BlockingRecv error: recv got EOF from 127.8.204.130:54733 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 22:54:17 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:17 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:17.498071 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.488951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:56813)
0927 22:54:17.489503 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.489525 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:17.489703 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:17.490120 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:17.490127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:17.490564 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:17.490816 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:17.490831 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.491878 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.491886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:17.492820 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:17.492831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.493524 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.493530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:17.493701 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.494329 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:17.494352 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:17.497745 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
W20260927 22:54:17.498167 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:17.489251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:56813 (local address 127.8.204.190:45041)
0927 22:54:17.489439 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:17.489446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:17.489642 (+ 196us) server_negotiation.cc:407] Connection header received
0927 22:54:17.489841 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:17.489847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:17.489935 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:17.490050 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:17.490973 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.491745 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.492965 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.493373 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.493504 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.497927 (+ 4423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:56813: BlockingRecv error: recv got EOF from 127.8.204.131:56813 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 22:54:17 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:17 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:17.734139 18274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:17.725185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:40333)
0927 22:54:17.725739 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:17.725787 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:17.725926 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:17.726335 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:17.726340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:17.726911 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:17.727130 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:17.727142 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.728253 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.728256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:17.729156 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:17.729166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.729828 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.729832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:17.729960 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.730536 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:17.730552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:17.733901 (+ 3349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":305,"threads_started":1}
W20260927 22:54:17.734330 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:17.725221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:40333 (local address 127.8.204.190:45041)
0927 22:54:17.725460 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:17.725470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:17.725880 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:54:17.726085 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:17.726093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:17.726174 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:17.726270 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:17.727298 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.728135 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.729280 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:17.729711 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:17.729800 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:17.734088 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:40333: BlockingRecv error: recv got EOF from 127.8.204.129:40333 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 22:54:18 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:18 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:18.016755 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:18.007871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:39999)
0927 22:54:18.008430 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:18.008445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:18.008572 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:18.008857 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:18.008860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:18.009250 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:18.009411 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:18.009420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.010341 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.010344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:18.011318 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:18.011344 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.012012 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.012016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:18.012142 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:18.012713 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:18.012730 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:18.016521 (+ 3791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":333,"threads_started":1}
W20260927 22:54:18.016927 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:18.007921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:39999 (local address 127.8.204.190:45041)
0927 22:54:18.008083 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:18.008087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:18.008531 (+ 444us) server_negotiation.cc:407] Connection header received
0927 22:54:18.008663 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:18.008667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:18.008733 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:18.008814 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:18.009528 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.010232 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.011478 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.011887 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.011988 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:18.016696 (+ 4708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:39999: BlockingRecv error: recv got EOF from 127.8.204.130:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:54:18 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:18 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:18.507705 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:18.498927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:33725)
0927 22:54:18.499475 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:18.499505 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:18.499619 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:18.499979 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:18.499984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:18.500381 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:18.500627 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:18.500638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.501731 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.501737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:18.502675 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:18.502687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.503448 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.503453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:18.503629 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:18.504197 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:18.504214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:18.507450 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":259,"threads_started":1}
W20260927 22:54:18.507893 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:18.499037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:33725 (local address 127.8.204.190:45041)
0927 22:54:18.499255 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:18.499261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:18.499586 (+ 325us) server_negotiation.cc:407] Connection header received
0927 22:54:18.499724 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:18.499729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:18.499805 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:18.499929 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:18.500765 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.501583 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.502817 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.503227 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.503421 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:18.507671 (+ 4250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:33725: BlockingRecv error: recv got EOF from 127.8.204.131:33725 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:54:18 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:18 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:18.743697 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:18.735102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:58807)
0927 22:54:18.735823 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:18.735839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:18.735948 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:18.736357 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:18.736361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:18.736846 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:18.737034 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:18.737049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.737980 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.737983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:18.738859 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:18.738869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.739500 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.739504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:18.739620 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:18.740164 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:18.740180 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:18.743474 (+ 3294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":498,"threads_started":1}
W20260927 22:54:18.743844 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:18.735258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:58807 (local address 127.8.204.190:45041)
0927 22:54:18.735529 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:18.735539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:18.735920 (+ 381us) server_negotiation.cc:407] Connection header received
0927 22:54:18.736092 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:18.736097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:18.736193 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:18.736290 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:18.737180 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.737868 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.738970 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:18.739371 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:18.739453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:18.743634 (+ 4181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:58807: BlockingRecv error: recv got EOF from 127.8.204.129:58807 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:54:19 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:19 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:19.026314 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:19.017762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:37031)
0927 22:54:19.018201 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:19.018217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:19.018326 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:19.018609 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:19.018612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:19.018971 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:19.019114 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:19.019122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.020088 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.020091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:19.020971 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:19.020983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.021629 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.021634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:19.021754 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:19.022299 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:19.022315 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:19.026071 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":203,"threads_started":1}
W20260927 22:54:19.026521 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:19.017980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:37031 (local address 127.8.204.190:45041)
0927 22:54:19.018204 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:19.018209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:19.018289 (+ 80us) server_negotiation.cc:407] Connection header received
0927 22:54:19.018418 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:19.018421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:19.018482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:19.018561 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:19.019228 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.019964 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.021094 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.021507 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.021598 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:19.026252 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:37031: BlockingRecv error: recv got EOF from 127.8.204.130:37031 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 22:54:19 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:19 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.131@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:19.517623 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:19.508770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.131:54681)
0927 22:54:19.509231 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:19.509246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:19.509404 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:19.509731 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:19.509736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:19.510140 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:19.510286 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:19.510294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.511330 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.511335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:19.512259 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:19.512268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.512957 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.512963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:19.513134 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:19.513762 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:19.513778 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:19.517392 (+ 3614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
W20260927 22:54:19.517782 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:19.508839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:54681 (local address 127.8.204.190:45041)
0927 22:54:19.509086 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:19.509092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:19.509334 (+ 242us) server_negotiation.cc:407] Connection header received
0927 22:54:19.509546 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:19.509550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:19.509607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:19.509706 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:19.510430 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.511166 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.512393 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.512816 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.512948 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:19.517572 (+ 4624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.131:54681: BlockingRecv error: recv got EOF from 127.8.204.131:54681 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 22:54:19 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:19 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.129@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:19.757899 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:19.744916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.129:48973)
0927 22:54:19.745487 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:19.745504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:19.745608 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:19.745894 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:19.745897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:19.746264 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:19.746444 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:19.746452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.747452 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.747455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:19.748340 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:19.748350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.749133 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.749137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:19.749256 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:19.754460 (+ 5204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:19.754479 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:19.757663 (+ 3184us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":230,"threads_started":1}
W20260927 22:54:19.758073 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:19.745079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:48973 (local address 127.8.204.190:45041)
0927 22:54:19.745417 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:19.745424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:19.745579 (+ 155us) server_negotiation.cc:407] Connection header received
0927 22:54:19.745702 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:19.745705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:19.745763 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:19.745851 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:19.746576 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.747334 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.748543 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:19.749006 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:19.749106 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:19.757843 (+ 8737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.129:48973: BlockingRecv error: recv got EOF from 127.8.204.129:48973 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Sep 27 22:54:20 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:54:20 dist-test-slave-40vx krb5kdc[17234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.204.130@KRBTEST.COM for oryx/127.8.204.190@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:54:20.035686 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.027457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:45041 (local address 127.8.204.130:45535)
0927 22:54:20.027935 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:20.027951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:20.028055 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:20.028361 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:20.028364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:20.028701 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:20.028875 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.028883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.029825 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.029829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.030719 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:20.030730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.031369 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.031373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:20.031508 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.032048 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:20.032064 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:20.035324 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.204.190:45041: Server oryx/127.8.204.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":258,"threads_started":1}
W20260927 22:54:20.035756 18236 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:20.027495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:45535 (local address 127.8.204.190:45041)
0927 22:54:20.027650 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.027654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.028017 (+ 363us) server_negotiation.cc:407] Connection header received
0927 22:54:20.028142 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.028145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.028202 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.028298 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.028994 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.029712 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.030830 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.031194 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.031299 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.035504 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.204.130:45535: BlockingRecv error: recv got EOF from 127.8.204.130:45535 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:54:20.154747 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17821
I20260927 22:54:20.166963 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.155971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.8.204.129:43699)
0927 22:54:20.156677 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.156683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.156712 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:54:20.156769 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.156775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.157028 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.157187 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.158526 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.159707 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.160830 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.161369 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.161465 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.163842 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.163850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.163864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.163897 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.166463 (+ 2566us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.204.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:54:20.166741 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53738: Request ticket server kudu/127.8.204.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":359,"threads_started":1}
W20260927 22:54:20.167093 18283 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53738: Request ticket server kudu/127.8.204.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:54:20.167090 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17821: Not authorized: Client connection negotiation failed: client connection to 127.8.204.129:43699: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.204.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:54:20.167418 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17821
I20260927 22:54:20.184443 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17972
I20260927 22:54:20.197216 18284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.185632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.8.204.130:39981)
0927 22:54:20.186202 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.186209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.186239 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:54:20.186308 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.186315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.186738 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.186955 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.188361 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.189401 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.190637 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.191213 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.191380 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.193862 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.193871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.193885 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.193922 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.196535 (+ 2613us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.204.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:54:20.196875 (+ 340us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35718: Request ticket server kudu/127.8.204.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
W20260927 22:54:20.197283 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17972: Not authorized: Client connection negotiation failed: client connection to 127.8.204.130:39981: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.204.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:54:20.197327 18284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35718: Request ticket server kudu/127.8.204.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:54:20.197420 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17972
I20260927 22:54:20.213654 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18105
I20260927 22:54:20.226008 18285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.214681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48674 (local address 127.8.204.131:33929)
0927 22:54:20.215265 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.215296 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.215329 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:54:20.215409 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.215418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.215727 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.215953 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.217178 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.218661 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.219913 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.220462 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.220572 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.222992 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.223001 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.223016 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:20.223070 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.225504 (+ 2434us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.204.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:54:20.225774 (+ 270us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48674: Request ticket server kudu/127.8.204.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":281,"threads_started":1}
W20260927 22:54:20.226128 18285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48674: Request ticket server kudu/127.8.204.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:54:20.226138 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18105: Not authorized: Client connection negotiation failed: client connection to 127.8.204.131:33929: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.204.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:54:20.226303 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18105
I20260927 22:54:20.242698 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 17752
I20260927 22:54:20.244087 17814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:20.359539 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 17752
2026-09-27T22:54:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21018 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:20Z Disabled control of system clock
I20260927 22:54:20.433363 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40241
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:54:20.748571 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:20.748844 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:20.748921 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:20.756039 18290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:20.756134 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:20.756170 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:20.756227 18290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:20.756270 18290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:20.765141 18290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:20.766377 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:20.768074 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:20.780043 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:20.780094 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:20.780021 18298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:20.780370 18290 server_base.cc:1034] running on GCE node
I20260927 22:54:20.781356 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:20.782701 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:20.783893 18290 hybrid_clock.cc:648] HybridClock initialized: now 1790549660783842 us; error 37 us; skew 500 ppm
I20260927 22:54:20.784188 18290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:20.790614 18290 webserver.cc:460] Webserver started at http://127.8.204.190:34493/ using document root <none> and password file <none>
I20260927 22:54:20.791313 18290 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:20.791414 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:20.791689 18290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:20.794492 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "5765b83f36be4c46a9889bff0e14714c"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-40vx"
server_key: "6e8007e29d09d3cd0d1ea4580b8c6273"
server_key_iv: "bc3986dc5eb9481afcd291151ca48995"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.795250 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "5765b83f36be4c46a9889bff0e14714c"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-40vx"
server_key: "6e8007e29d09d3cd0d1ea4580b8c6273"
server_key_iv: "bc3986dc5eb9481afcd291151ca48995"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.800318 18290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:54:20.803539 18304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:20.804781 18290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:20.804952 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "5765b83f36be4c46a9889bff0e14714c"
format_stamp: "Formatted at 2026-09-27 22:54:20 on dist-test-slave-40vx"
server_key: "6e8007e29d09d3cd0d1ea4580b8c6273"
server_key_iv: "bc3986dc5eb9481afcd291151ca48995"
server_key_version: "encryptionkey@0"
I20260927 22:54:20.805186 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:20.828397 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:20.829274 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:20.829512 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:20.846894 18290 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40241
I20260927 22:54:20.846917 18355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40241 every 8 connection(s)
I20260927 22:54:20.848224 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:20.851768 18356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:20.853053 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18290
I20260927 22:54:20.853240 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:54:20.854036 9010 external_mini_cluster.cc:1428] Setting key 44aa2dc8b723f9e727348e7221a64859
I20260927 22:54:20.868191 18356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c: Bootstrap starting.
I20260927 22:54:20.872507 18356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:20.873940 18356 log.cc:824] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:20.876160 18359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:20.857518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.8.204.190:40241)
0927 22:54:20.860061 (+ 2543us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:20.860075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:20.860125 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:54:20.862887 (+ 2762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:20.862920 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:20.863760 (+ 840us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:20.864264 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:20.864910 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.867215 (+ 2305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.869919 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:20.871240 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:20.871521 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:20.872795 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:20.872807 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:20.872832 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:20.873558 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:20.873654 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:20.873698 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:20.873801 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:20.874403 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 22:54:20.874768 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2052,"thread_start_us":1845,"threads_started":1}
I20260927 22:54:20.877417 18356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c: No bootstrap required, opened a new log
I20260927 22:54:20.888556 18356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5765b83f36be4c46a9889bff0e14714c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40241 } }
I20260927 22:54:20.889014 18356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:20.889138 18356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5765b83f36be4c46a9889bff0e14714c, State: Initialized, Role: FOLLOWER
I20260927 22:54:20.889772 18356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5765b83f36be4c46a9889bff0e14714c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40241 } }
I20260927 22:54:20.890017 18356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:20.890158 18356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:20.890369 18356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:20.892620 18356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5765b83f36be4c46a9889bff0e14714c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40241 } }
I20260927 22:54:20.893193 18356 leader_election.cc:302] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5765b83f36be4c46a9889bff0e14714c; no voters:
I20260927 22:54:20.893857 18356 leader_election.cc:288] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:20.894012 18361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:20.894851 18361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [term 1 LEADER]: Becoming Leader. State: Replica: 5765b83f36be4c46a9889bff0e14714c, State: Running, Role: LEADER
I20260927 22:54:20.895514 18361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5765b83f36be4c46a9889bff0e14714c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40241 } }
I20260927 22:54:20.896183 18356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:20.898741 18363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5765b83f36be4c46a9889bff0e14714c. Latest consensus state: current_term: 1 leader_uuid: "5765b83f36be4c46a9889bff0e14714c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5765b83f36be4c46a9889bff0e14714c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40241 } } }
I20260927 22:54:20.899132 18363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:20.898847 18362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5765b83f36be4c46a9889bff0e14714c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5765b83f36be4c46a9889bff0e14714c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40241 } } }
I20260927 22:54:20.900094 18362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:20.904156 18366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:20.910507 18366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:20.920503 18366 catalog_manager.cc:1347] Generated new cluster ID: 2299ef56b4d84ee18cd7711b993cb054
I20260927 22:54:20.920612 18366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:20.946192 18366 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:20.948081 18366 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:20.959000 18366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5765b83f36be4c46a9889bff0e14714c: Generated new TSK 0
I20260927 22:54:20.959973 18366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:54:20.977077 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40241
--builtin_ntp_servers=127.8.204.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:54:21.298763 18380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:21.299012 18380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:21.299166 18380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:21.306559 18380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:21.306658 18380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:21.306818 18380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:54:21.315814 18380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:21.317173 18380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:21.318933 18380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:21.331808 18385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:21.331761 18388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:21.331843 18386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:21.332715 18380 server_base.cc:1034] running on GCE node
I20260927 22:54:21.333321 18380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:21.334259 18380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:21.335469 18380 hybrid_clock.cc:648] HybridClock initialized: now 1790549661335405 us; error 55 us; skew 500 ppm
I20260927 22:54:21.335789 18380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:21.338004 18380 webserver.cc:460] Webserver started at http://127.8.204.129:45703/ using document root <none> and password file <none>
I20260927 22:54:21.338666 18380 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:21.338801 18380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:21.339125 18380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:21.341859 18380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "e147fd21448d436097a5459f918e384c"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-40vx"
server_key: "d67919a17d8c5b1765f677b68c467c6b"
server_key_iv: "ce1e8450cd964e0f718153df82673890"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.342660 18380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "e147fd21448d436097a5459f918e384c"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-40vx"
server_key: "d67919a17d8c5b1765f677b68c467c6b"
server_key_iv: "ce1e8450cd964e0f718153df82673890"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.347474 18380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 22:54:21.350674 18394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.351969 18380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:21.352155 18380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "e147fd21448d436097a5459f918e384c"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-40vx"
server_key: "d67919a17d8c5b1765f677b68c467c6b"
server_key_iv: "ce1e8450cd964e0f718153df82673890"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.352365 18380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:21.366588 18380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:21.367443 18380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:21.367754 18380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:21.368695 18380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:21.370138 18380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:21.370254 18380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.370390 18380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:21.370486 18380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.402938 18380 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:44385
I20260927 22:54:21.402980 18506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:44385 every 8 connection(s)
I20260927 22:54:21.404302 18380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:21.408393 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18380
I20260927 22:54:21.408589 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:21.409243 9010 external_mini_cluster.cc:1428] Setting key fc53338b57a6713d4fdc5d9ca66c5641
I20260927 22:54:21.411973 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40241
--builtin_ntp_servers=127.8.204.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:54:21.417526 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.406812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:47393 (local address 127.8.204.190:40241)
0927 22:54:21.407212 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.407216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.408337 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 22:54:21.410431 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.410435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.410505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.410593 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.411888 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.413007 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.414493 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.415184 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.415356 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.416578 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.416586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.416594 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:21.416617 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.416668 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:21.416685 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.416779 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.416973 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.417077 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":306,"thread_start_us":221,"threads_started":1}
I20260927 22:54:21.418219 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.407293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40241 (local address 127.8.204.129:47393)
0927 22:54:21.408142 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.408234 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.410177 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.410872 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.410882 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.411005 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.411715 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.411730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.413154 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.413157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.414349 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.414360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.415427 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.415432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.415616 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.416290 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:54:21.416293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.416330 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:54:21.416398 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.416779 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.416786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.416791 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.416916 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.417201 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":246,"threads_started":1}
I20260927 22:54:21.419724 18507 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40241
I20260927 22:54:21.420030 18507 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:21.420861 18507 heartbeater.cc:511] Master 127.8.204.190:40241 requested a full tablet report, sending...
I20260927 22:54:21.423022 18321 ts_manager.cc:194] Registered new tserver with Master: e147fd21448d436097a5459f918e384c (127.8.204.129:44385)
I20260927 22:54:21.425203 18321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.129:47393
W20260927 22:54:21.731082 18511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:21.731421 18511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:21.731513 18511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:21.738445 18511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:21.738529 18511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:21.738626 18511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:54:21.747236 18511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:21.748443 18511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:21.750092 18511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:21.762533 18519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:21.762529 18516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:21.762562 18517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:21.763381 18511 server_base.cc:1034] running on GCE node
I20260927 22:54:21.763993 18511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:21.764904 18511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:21.766100 18511 hybrid_clock.cc:648] HybridClock initialized: now 1790549661766054 us; error 45 us; skew 500 ppm
I20260927 22:54:21.766448 18511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:21.768647 18511 webserver.cc:460] Webserver started at http://127.8.204.130:41161/ using document root <none> and password file <none>
I20260927 22:54:21.769325 18511 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:21.769445 18511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:21.769774 18511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:21.772521 18511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "976df763607e4907913a0d8934a1a86b"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-40vx"
server_key: "2dc1ea8d1fd3b1dd2f49ec018888b560"
server_key_iv: "46f994f1e6173c5eef0a2a5ad5f97499"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.773280 18511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "976df763607e4907913a0d8934a1a86b"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-40vx"
server_key: "2dc1ea8d1fd3b1dd2f49ec018888b560"
server_key_iv: "46f994f1e6173c5eef0a2a5ad5f97499"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.778024 18511 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:54:21.781136 18525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.782389 18511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:54:21.782564 18511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "976df763607e4907913a0d8934a1a86b"
format_stamp: "Formatted at 2026-09-27 22:54:21 on dist-test-slave-40vx"
server_key: "2dc1ea8d1fd3b1dd2f49ec018888b560"
server_key_iv: "46f994f1e6173c5eef0a2a5ad5f97499"
server_key_version: "encryptionkey@0"
I20260927 22:54:21.782760 18511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:21.801043 18511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:21.801793 18511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:21.802078 18511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:21.802946 18511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:21.804309 18511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:21.804440 18511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.804554 18511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:21.804641 18511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:21.851884 18511 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:42873
I20260927 22:54:21.851936 18637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:42873 every 8 connection(s)
I20260927 22:54:21.853142 18511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:21.856750 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18511
I20260927 22:54:21.856927 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:21.857424 9010 external_mini_cluster.cc:1428] Setting key 07ebc0a735f99bf70563c62ba2a29f4a
I20260927 22:54:21.859956 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40241
--builtin_ntp_servers=127.8.204.148:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:54:21.868088 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.855768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:35909 (local address 127.8.204.190:40241)
0927 22:54:21.855972 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:21.855979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:21.857612 (+ 1633us) server_negotiation.cc:407] Connection header received
0927 22:54:21.860097 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:21.860104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:21.860194 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:21.860298 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:21.861570 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.862743 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.865006 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.865699 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.865842 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.867187 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:21.867191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:21.867195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:21.867211 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:21.867239 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:21.867250 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:21.867382 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:21.867629 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:54:21.867732 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:54:21.869025 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:21.856358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40241 (local address 127.8.204.130:35909)
0927 22:54:21.857458 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:21.857507 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:21.859752 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:21.860529 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:21.860540 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:21.860650 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:21.861392 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.861415 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.862898 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.862903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.864851 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:21.864872 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:21.865902 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:21.865906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:21.866101 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:21.866842 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:54:21.866847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:21.866899 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:54:21.867004 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:21.867396 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:21.867409 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:21.867418 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:54:21.867594 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 22:54:21.867906 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":324,"threads_started":1}
I20260927 22:54:21.870433 18638 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40241
I20260927 22:54:21.870720 18638 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:21.871537 18638 heartbeater.cc:511] Master 127.8.204.190:40241 requested a full tablet report, sending...
I20260927 22:54:21.872949 18321 ts_manager.cc:194] Registered new tserver with Master: 976df763607e4907913a0d8934a1a86b (127.8.204.130:42873)
I20260927 22:54:21.874145 18321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.130:35909
W20260927 22:54:22.180074 18641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:22.180347 18641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:22.180438 18641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:22.187901 18641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:22.187996 18641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:22.188097 18641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:54:22.196856 18641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:22.198143 18641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:22.199797 18641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:22.212571 18646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:22.212641 18647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:22.212662 18649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:22.213330 18641 server_base.cc:1034] running on GCE node
I20260927 22:54:22.214039 18641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:22.214949 18641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:22.216169 18641 hybrid_clock.cc:648] HybridClock initialized: now 1790549662216135 us; error 64 us; skew 500 ppm
I20260927 22:54:22.216485 18641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:22.218581 18641 webserver.cc:460] Webserver started at http://127.8.204.131:33837/ using document root <none> and password file <none>
I20260927 22:54:22.219254 18641 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:22.219434 18641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:22.219745 18641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:22.222501 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "1e6eb1c3383945f5b35a3d4e9ca652ae"
format_stamp: "Formatted at 2026-09-27 22:54:22 on dist-test-slave-40vx"
server_key: "c98a021a96bec1d6139b4089bf230e30"
server_key_iv: "c934393a3068ffafc23d1c455a2b96a7"
server_key_version: "encryptionkey@0"
I20260927 22:54:22.223265 18641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "1e6eb1c3383945f5b35a3d4e9ca652ae"
format_stamp: "Formatted at 2026-09-27 22:54:22 on dist-test-slave-40vx"
server_key: "c98a021a96bec1d6139b4089bf230e30"
server_key_iv: "c934393a3068ffafc23d1c455a2b96a7"
server_key_version: "encryptionkey@0"
I20260927 22:54:22.228093 18641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:54:22.231073 18655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:22.232473 18641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:54:22.232640 18641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "1e6eb1c3383945f5b35a3d4e9ca652ae"
format_stamp: "Formatted at 2026-09-27 22:54:22 on dist-test-slave-40vx"
server_key: "c98a021a96bec1d6139b4089bf230e30"
server_key_iv: "c934393a3068ffafc23d1c455a2b96a7"
server_key_version: "encryptionkey@0"
I20260927 22:54:22.232846 18641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:22.242661 18641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:22.243458 18641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:22.243736 18641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:22.244592 18641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:22.245895 18641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:22.246031 18641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:22.246148 18641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:22.246235 18641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:22.277943 18641 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:39497
I20260927 22:54:22.277990 18767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:39497 every 8 connection(s)
I20260927 22:54:22.279253 18641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:22.283125 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18641
I20260927 22:54:22.283304 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:22.283849 9010 external_mini_cluster.cc:1428] Setting key e3a02830bc94ebfc39b16aa39509241a
I20260927 22:54:22.291540 18509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:22.281740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:36867 (local address 127.8.204.190:40241)
0927 22:54:22.281912 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:22.281916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:22.283551 (+ 1635us) server_negotiation.cc:407] Connection header received
0927 22:54:22.285572 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:22.285577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:22.285664 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:22.285770 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:22.286940 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.287771 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.289211 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.289585 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.289674 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:22.290830 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:22.290835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:22.290838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:22.290854 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:22.290883 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:22.290893 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:22.290957 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:22.291146 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 22:54:22.291210 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:54:22.292497 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:22.282235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40241 (local address 127.8.204.131:36867)
0927 22:54:22.283385 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:22.283429 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:22.285273 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:22.285993 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:22.286001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:22.286132 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:22.286743 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:22.286768 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.287902 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.287906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:22.289098 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:22.289109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.289758 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.289763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:22.289949 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:22.290553 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:54:22.290555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:22.290590 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:54:22.290662 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:22.290972 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:22.290985 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:22.290990 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:54:22.291121 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 22:54:22.291491 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":328,"threads_started":1}
I20260927 22:54:22.293759 18768 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40241
I20260927 22:54:22.294009 18768 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:22.294624 18768 heartbeater.cc:511] Master 127.8.204.190:40241 requested a full tablet report, sending...
I20260927 22:54:22.295950 18321 ts_manager.cc:194] Registered new tserver with Master: 1e6eb1c3383945f5b35a3d4e9ca652ae (127.8.204.131:39497)
I20260927 22:54:22.296761 18321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.131:36867
I20260927 22:54:22.298904 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:54:22.304986 18509 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:22.303299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.8.204.190:40241)
0927 22:54:22.303452 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:22.303457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:22.303926 (+ 469us) server_negotiation.cc:407] Connection header received
0927 22:54:22.304143 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:22.304148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:22.304223 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:22.304320 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:22.304751 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45128: BlockingRecv error: recv got EOF from 127.0.0.1:45128 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:54:22.305903 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18380
I20260927 22:54:22.313099 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:22.306742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.8.204.129:44385)
0927 22:54:22.307235 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:22.307243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:22.307335 (+ 92us) server_negotiation.cc:407] Connection header received
0927 22:54:22.307404 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:22.307410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:22.307594 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:22.307716 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:22.308045 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.309033 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.310274 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.310809 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.310905 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:22.311852 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:22.311859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:22.311871 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:22.312446 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:22.312498 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:22.312516 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:22.312602 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:22.312699 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 22:54:22.312810 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":268,"threads_started":1}
I20260927 22:54:22.314253 18502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:22.394685 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18380
I20260927 22:54:22.410830 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18511
I20260927 22:54:22.418529 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:22.411695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.8.204.130:42873)
0927 22:54:22.412193 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:22.412201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:22.412226 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:54:22.412285 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:22.412291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:22.412520 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:22.412689 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:22.413075 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.414059 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.415388 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.415935 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.416079 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:22.417075 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:22.417083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:22.417097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:22.417702 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:22.417748 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:22.417766 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:22.417890 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:22.418044 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:54:22.418186 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
I20260927 22:54:22.419845 18633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:22.499495 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18511
I20260927 22:54:22.515226 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18641
I20260927 22:54:22.522792 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:22.516170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.8.204.131:39497)
0927 22:54:22.516770 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:22.516777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:22.516806 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:54:22.516871 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:22.516877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:22.517089 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:22.517209 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:22.517599 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.518617 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.519877 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:22.520406 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:22.520511 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:22.521455 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:22.521462 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:22.521483 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:22.522039 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:22.522082 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:22.522099 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:22.522187 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:22.522303 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 22:54:22.522433 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":304,"threads_started":1}
I20260927 22:54:22.524050 18763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:22.604524 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18641
I20260927 22:54:22.620137 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18290
I20260927 22:54:22.621277 18351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:22.720963 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18290
2026-09-27T22:54:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2357 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:22Z Disabled control of system clock
I20260927 22:54:22.793422 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:32973
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:32973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:54:23.108898 18787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:23.109190 18787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:23.109272 18787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:23.116312 18787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:23.116413 18787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:23.116451 18787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:23.116513 18787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:23.116555 18787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:23.125501 18787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:32973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:32973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:23.126740 18787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:23.128470 18787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:23.140316 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:23.140372 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:23.140343 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:23.141268 18787 server_base.cc:1034] running on GCE node
I20260927 22:54:23.142114 18787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:23.143548 18787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:23.144770 18787 hybrid_clock.cc:648] HybridClock initialized: now 1790549663144734 us; error 48 us; skew 500 ppm
I20260927 22:54:23.145112 18787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:23.147357 18787 webserver.cc:460] Webserver started at http://127.8.204.190:37679/ using document root <none> and password file <none>
I20260927 22:54:23.148047 18787 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:23.148173 18787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:23.148504 18787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:23.152629 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "7d28c5fedd184b98aa707c79b55d2ffc"
format_stamp: "Formatted at 2026-09-27 22:54:23 on dist-test-slave-40vx"
server_key: "bc7bb5fc2bb5bac78dfb6426cf82185d"
server_key_iv: "8551bb30cdfe6bf258d11abbc273d899"
server_key_version: "encryptionkey@0"
I20260927 22:54:23.153389 18787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "7d28c5fedd184b98aa707c79b55d2ffc"
format_stamp: "Formatted at 2026-09-27 22:54:23 on dist-test-slave-40vx"
server_key: "bc7bb5fc2bb5bac78dfb6426cf82185d"
server_key_iv: "8551bb30cdfe6bf258d11abbc273d899"
server_key_version: "encryptionkey@0"
I20260927 22:54:23.158176 18787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:54:23.161389 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:23.162613 18787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:54:23.162781 18787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "7d28c5fedd184b98aa707c79b55d2ffc"
format_stamp: "Formatted at 2026-09-27 22:54:23 on dist-test-slave-40vx"
server_key: "bc7bb5fc2bb5bac78dfb6426cf82185d"
server_key_iv: "8551bb30cdfe6bf258d11abbc273d899"
server_key_version: "encryptionkey@0"
I20260927 22:54:23.163002 18787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:23.168468 18787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:23.169221 18787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:23.169518 18787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:23.187525 18787 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:32973
I20260927 22:54:23.187544 18852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:32973 every 8 connection(s)
I20260927 22:54:23.188827 18787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:23.192188 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18787
I20260927 22:54:23.192411 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:54:23.192715 18853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:23.193202 9010 external_mini_cluster.cc:1428] Setting key 96519fd6019f90eda7d14e0ce5a83277
I20260927 22:54:23.201776 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:23.195200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56952 (local address 127.8.204.190:32973)
0927 22:54:23.196285 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:23.196295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:23.196335 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:54:23.198464 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:23.198486 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:23.198902 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:23.199237 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:23.200280 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:23.200286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:23.200298 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:23.200310 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:23.200365 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:23.200379 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:23.200527 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:23.200645 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 22:54:23.200917 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":418,"threads_started":1}
I20260927 22:54:23.205492 18853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc: Bootstrap starting.
I20260927 22:54:23.208282 18853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:23.209321 18853 log.cc:824] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:23.212009 18853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc: No bootstrap required, opened a new log
I20260927 22:54:23.219166 18853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 32973 } }
I20260927 22:54:23.219630 18853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:23.219736 18853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d28c5fedd184b98aa707c79b55d2ffc, State: Initialized, Role: FOLLOWER
I20260927 22:54:23.220350 18853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 32973 } }
I20260927 22:54:23.220580 18853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:23.220721 18853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:23.220887 18853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:23.222961 18853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 32973 } }
I20260927 22:54:23.223518 18853 leader_election.cc:302] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d28c5fedd184b98aa707c79b55d2ffc; no voters:
I20260927 22:54:23.224217 18853 leader_election.cc:288] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:23.224341 18858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:23.225072 18858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [term 1 LEADER]: Becoming Leader. State: Replica: 7d28c5fedd184b98aa707c79b55d2ffc, State: Running, Role: LEADER
I20260927 22:54:23.225682 18858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 32973 } }
I20260927 22:54:23.226302 18853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:23.228420 18860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d28c5fedd184b98aa707c79b55d2ffc. Latest consensus state: current_term: 1 leader_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 32973 } } }
I20260927 22:54:23.228832 18860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:23.230368 18867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:23.230319 18859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d28c5fedd184b98aa707c79b55d2ffc" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 32973 } } }
I20260927 22:54:23.230973 18859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:23.235181 18867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:23.242714 18867 catalog_manager.cc:1347] Generated new cluster ID: 523f8f62237a4c40bde3131b90f638dc
I20260927 22:54:23.242810 18867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:23.259737 18867 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:23.260213 18867 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:23.267042 18867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d28c5fedd184b98aa707c79b55d2ffc: Generated new TSK 0
I20260927 22:54:23.267848 18867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:54:23.276366 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:32973
--builtin_ntp_servers=127.8.204.148:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:54:23.597443 18877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:23.597704 18877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:23.597851 18877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:23.604825 18877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:23.604911 18877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:23.605093 18877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:54:23.614188 18877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:32973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:23.615522 18877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:23.617237 18877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:23.629984 18882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:23.629946 18883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:23.629904 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:23.630878 18877 server_base.cc:1034] running on GCE node
I20260927 22:54:23.631492 18877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:23.632370 18877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:23.633590 18877 hybrid_clock.cc:648] HybridClock initialized: now 1790549663633551 us; error 49 us; skew 500 ppm
I20260927 22:54:23.633906 18877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:23.636224 18877 webserver.cc:460] Webserver started at http://127.8.204.129:45859/ using document root <none> and password file <none>
I20260927 22:54:23.636914 18877 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:23.637049 18877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:23.637373 18877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:23.640192 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "97a9f0b7d035470e9e8a266fdd884f72"
format_stamp: "Formatted at 2026-09-27 22:54:23 on dist-test-slave-40vx"
server_key: "565834634880eee5aed8979f757d8f37"
server_key_iv: "394a99c7c6a032e3cfe0dd69ad7b2e59"
server_key_version: "encryptionkey@0"
I20260927 22:54:23.640964 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "97a9f0b7d035470e9e8a266fdd884f72"
format_stamp: "Formatted at 2026-09-27 22:54:23 on dist-test-slave-40vx"
server_key: "565834634880eee5aed8979f757d8f37"
server_key_iv: "394a99c7c6a032e3cfe0dd69ad7b2e59"
server_key_version: "encryptionkey@0"
I20260927 22:54:23.645946 18877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:54:23.649290 18891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:23.650542 18877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:23.650712 18877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "97a9f0b7d035470e9e8a266fdd884f72"
format_stamp: "Formatted at 2026-09-27 22:54:23 on dist-test-slave-40vx"
server_key: "565834634880eee5aed8979f757d8f37"
server_key_iv: "394a99c7c6a032e3cfe0dd69ad7b2e59"
server_key_version: "encryptionkey@0"
I20260927 22:54:23.650913 18877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:23.656297 18877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:23.657051 18877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:23.657305 18877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:23.658151 18877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:23.659514 18877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:23.659617 18877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:23.659734 18877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:23.659827 18877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:23.690266 18877 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:33701
I20260927 22:54:23.690315 19003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:33701 every 8 connection(s)
I20260927 22:54:23.691520 18877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:23.699014 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:23.694057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:37333 (local address 127.8.204.190:32973)
0927 22:54:23.694255 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:23.694261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:23.695680 (+ 1419us) server_negotiation.cc:407] Connection header received
0927 22:54:23.697223 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:23.697226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:23.697276 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:23.697359 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:23.698467 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:23.698471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:23.698475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:23.698491 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:23.698518 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:23.698530 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:23.698619 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:23.698774 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 22:54:23.698819 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:54:23.699487 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 18877
I20260927 22:54:23.699688 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:23.700289 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:23.694580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:32973 (local address 127.8.204.129:37333)
0927 22:54:23.695537 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:23.695574 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:23.697000 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:23.697545 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:23.697553 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:23.697637 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:23.698213 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:54:23.698215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:23.698250 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:54:23.698327 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:23.698610 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:23.698616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:23.698621 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:54:23.698817 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 22:54:23.699058 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
I20260927 22:54:23.700436 9010 external_mini_cluster.cc:1428] Setting key 7c721e4962aac4cf84f2bdb55f57a51d
I20260927 22:54:23.701828 19004 heartbeater.cc:347] Connected to a master server at 127.8.204.190:32973
I20260927 22:54:23.702116 19004 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:23.702641 19004 heartbeater.cc:511] Master 127.8.204.190:32973 requested a full tablet report, sending...
I20260927 22:54:23.703207 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:32973
--builtin_ntp_servers=127.8.204.148:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:54:23.704680 18818 ts_manager.cc:194] Registered new tserver with Master: 97a9f0b7d035470e9e8a266fdd884f72 (127.8.204.129:33701)
W20260927 22:54:24.027604 19007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:24.027848 19007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:24.027937 19007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:24.035107 19007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:24.035202 19007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:24.035323 19007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:54:24.044446 19007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:32973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:24.045636 19007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:24.047385 19007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:24.060072 19012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:24.060393 19013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:24.060989 19015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:24.063596 19007 server_base.cc:1034] running on GCE node
I20260927 22:54:24.064373 19007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:24.065618 19007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:24.066916 19007 hybrid_clock.cc:648] HybridClock initialized: now 1790549664066771 us; error 100 us; skew 500 ppm
I20260927 22:54:24.067366 19007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:24.069638 19007 webserver.cc:460] Webserver started at http://127.8.204.130:41949/ using document root <none> and password file <none>
I20260927 22:54:24.070314 19007 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:24.070431 19007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:24.070766 19007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:24.073546 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "5938d070eca54db1935c70346d2b1459"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-40vx"
server_key: "5a0dee298bd96a2896a92b1d4217a58d"
server_key_iv: "71d2bb8e316314ff821ac4c80eb19e87"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.074299 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "5938d070eca54db1935c70346d2b1459"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-40vx"
server_key: "5a0dee298bd96a2896a92b1d4217a58d"
server_key_iv: "71d2bb8e316314ff821ac4c80eb19e87"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.079051 19007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:54:24.082383 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.083751 19007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:24.083926 19007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "5938d070eca54db1935c70346d2b1459"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-40vx"
server_key: "5a0dee298bd96a2896a92b1d4217a58d"
server_key_iv: "71d2bb8e316314ff821ac4c80eb19e87"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.084127 19007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:24.089473 19007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:24.090220 19007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:24.090479 19007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:24.091383 19007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:24.092698 19007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:24.092828 19007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.092948 19007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:24.093036 19007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.125686 19007 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38349
I20260927 22:54:24.125731 19133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38349 every 8 connection(s)
I20260927 22:54:24.127126 19007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:24.134332 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 19007
I20260927 22:54:24.134532 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:24.134938 9010 external_mini_cluster.cc:1428] Setting key 7027c403a1f34002bc830137683d8fa7
I20260927 22:54:24.134974 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.129707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:37663 (local address 127.8.204.190:32973)
0927 22:54:24.129902 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.129908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.131267 (+ 1359us) server_negotiation.cc:407] Connection header received
0927 22:54:24.132907 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.132910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.132960 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.133055 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.134250 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:24.134256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:24.134260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:24.134279 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:24.134316 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:24.134328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:24.134454 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:24.134637 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 22:54:24.134702 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:54:24.136181 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.130240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:32973 (local address 127.8.204.130:37663)
0927 22:54:24.131116 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:24.131158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:24.132673 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:24.133261 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:24.133268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:24.133357 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:24.133970 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:54:24.133972 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:24.134009 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:54:24.134085 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:24.134428 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:24.134438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:24.134445 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:54:24.134716 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 22:54:24.134974 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":259,"threads_started":1}
I20260927 22:54:24.137508 19134 heartbeater.cc:347] Connected to a master server at 127.8.204.190:32973
I20260927 22:54:24.137602 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:32973
--builtin_ntp_servers=127.8.204.148:35575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:54:24.137825 19134 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:24.138370 19134 heartbeater.cc:511] Master 127.8.204.190:32973 requested a full tablet report, sending...
I20260927 22:54:24.139698 18818 ts_manager.cc:194] Registered new tserver with Master: 5938d070eca54db1935c70346d2b1459 (127.8.204.130:38349)
W20260927 22:54:24.453166 19137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:24.453421 19137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:24.453506 19137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:24.460485 19137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:24.460570 19137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:24.460671 19137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:54:24.469305 19137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:32973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:24.470470 19137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:24.472133 19137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:24.484483 19142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:24.484534 19145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:24.484572 19143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:24.485311 19137 server_base.cc:1034] running on GCE node
I20260927 22:54:24.485806 19137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:24.486632 19137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:24.487834 19137 hybrid_clock.cc:648] HybridClock initialized: now 1790549664487764 us; error 58 us; skew 500 ppm
I20260927 22:54:24.488144 19137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:54:24.490334 19137 webserver.cc:460] Webserver started at http://127.8.204.131:42415/ using document root <none> and password file <none>
I20260927 22:54:24.490942 19137 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:24.491032 19137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:24.491405 19137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:24.494091 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "6f1f9537747f47bcb4107087ca761033"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-40vx"
server_key: "b107ca6a99125d284cfebce0c6fa8214"
server_key_iv: "b2757f82329933055799c87d210de3a6"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.494788 19137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "6f1f9537747f47bcb4107087ca761033"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-40vx"
server_key: "b107ca6a99125d284cfebce0c6fa8214"
server_key_iv: "b2757f82329933055799c87d210de3a6"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.499717 19137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260927 22:54:24.502867 19151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.504208 19137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:54:24.504374 19137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "6f1f9537747f47bcb4107087ca761033"
format_stamp: "Formatted at 2026-09-27 22:54:24 on dist-test-slave-40vx"
server_key: "b107ca6a99125d284cfebce0c6fa8214"
server_key_iv: "b2757f82329933055799c87d210de3a6"
server_key_version: "encryptionkey@0"
I20260927 22:54:24.504576 19137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:24.510047 19137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:24.510731 19137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:24.510983 19137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:24.511909 19137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:24.513217 19137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:24.513324 19137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.513448 19137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:24.513542 19137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:24.545270 19137 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:44965
I20260927 22:54:24.545346 19263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:44965 every 8 connection(s)
I20260927 22:54:24.546552 19137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:24.548419 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 19137
I20260927 22:54:24.548612 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:24.549108 9010 external_mini_cluster.cc:1428] Setting key 9b2de040b338770266d496caecd0a83e
I20260927 22:54:24.554998 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.549307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:46463 (local address 127.8.204.190:32973)
0927 22:54:24.549525 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.549532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.551154 (+ 1622us) server_negotiation.cc:407] Connection header received
0927 22:54:24.553040 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.553044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.553093 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.553173 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.554451 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:24.554455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:24.554458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:24.554473 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:24.554502 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:24.554514 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:24.554611 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:24.554765 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 22:54:24.554810 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:54:24.555977 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.549864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:32973 (local address 127.8.204.131:46463)
0927 22:54:24.551003 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:24.551048 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:24.552806 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:24.553369 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:24.553377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:24.553491 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:24.554131 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:54:24.554139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:24.554176 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:54:24.554316 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:24.554600 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:24.554607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:24.554612 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:54:24.554813 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 22:54:24.555001 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":366,"threads_started":1}
I20260927 22:54:24.557209 19264 heartbeater.cc:347] Connected to a master server at 127.8.204.190:32973
I20260927 22:54:24.557554 19264 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:24.558143 19264 heartbeater.cc:511] Master 127.8.204.190:32973 requested a full tablet report, sending...
I20260927 22:54:24.559494 18818 ts_manager.cc:194] Registered new tserver with Master: 6f1f9537747f47bcb4107087ca761033 (127.8.204.131:44965)
I20260927 22:54:24.564378 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:54:24.570030 18856 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:54:24.568662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.8.204.190:32973)
0927 22:54:24.568808 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.568812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.569317 (+ 505us) server_negotiation.cc:407] Connection header received
0927 22:54:24.569442 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.569445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.569497 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.569574 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.569842 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56960: BlockingRecv error: recv got EOF from 127.0.0.1:56960 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:54:24.570582 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18877
I20260927 22:54:24.573872 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.571535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.8.204.129:33701)
0927 22:54:24.572076 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.572083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.572106 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:54:24.572156 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.572162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.572361 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.572480 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.573300 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:24.573308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:24.573318 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:24.573336 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:24.573379 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:24.573392 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:24.573510 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:24.573587 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:54:24.573657 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":269,"threads_started":1}
I20260927 22:54:24.575004 18999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:24.656142 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18877
I20260927 22:54:24.671907 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 19007
I20260927 22:54:24.675467 19275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.672989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.8.204.130:38349)
0927 22:54:24.673471 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.673486 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.673511 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:54:24.673570 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.673576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.673783 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.673907 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.674769 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:24.674776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:24.674787 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:24.674798 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:24.674839 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:24.674851 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:24.674950 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:24.675102 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:54:24.675181 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":246,"threads_started":1}
I20260927 22:54:24.676662 19129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:24.756455 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 19007
I20260927 22:54:24.772235 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 19137
I20260927 22:54:24.779369 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:24.773220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.8.204.131:44965)
0927 22:54:24.773938 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:24.773949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:24.773999 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:54:24.774093 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:24.774104 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:24.774415 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:24.774599 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:24.778581 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:24.778588 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:24.778600 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:54:24.778621 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:24.778666 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:54:24.778680 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:24.778811 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:24.778956 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:54:24.779044 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":311,"threads_started":1}
I20260927 22:54:24.780602 19259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:24.862223 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 19137
I20260927 22:54:24.878327 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 18787
I20260927 22:54:24.879494 18848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:24.981652 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 18787
2026-09-27T22:54:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2259 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:54:25 dist-test-slave-40vx krb5kdc[19282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:54:25 dist-test-slave-40vx krb5kdc[19282](info): set up 2 sockets
Sep 27 22:54:25 dist-test-slave-40vx krb5kdc[19282](info): commencing operation
krb5kdc: starting...
W20260927 22:54:26.084318 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:54:26 dist-test-slave-40vx krb5kdc[19282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:54:26.299211 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:38587
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:38587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:26.614411 19295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:26.614753 19295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:26.614902 19295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:26.621825 19295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:54:26.621961 19295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:26.622073 19295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:26.622174 19295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:54:26.622249 19295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:54:26.631672 19295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:38587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:38587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:26.632899 19295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:26.634171 19295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:26.646174 19303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:26.646217 19300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:26.646265 19301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:26.648729 19295 server_base.cc:1034] running on GCE node
I20260927 22:54:26.649565 19295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:26.650887 19295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:26.652120 19295 hybrid_clock.cc:648] HybridClock initialized: now 1790549666652083 us; error 47 us; skew 500 ppm
I20260927 22:54:26.652458 19295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:26 dist-test-slave-40vx krb5kdc[19282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:26.655448 19295 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:54:26.656879 19295 webserver.cc:460] Webserver started at http://127.8.204.190:36459/ using document root <none> and password file <none>
I20260927 22:54:26.657554 19295 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:26.657697 19295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:26.657991 19295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:26.660853 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "56d96ff6639f45dca8d175be5545e447"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-40vx"
server_key: "4d20552cd661188ea31b6b6c94272328"
server_key_iv: "6125d91c112eb577723943390743e904"
server_key_version: "encryptionkey@0"
I20260927 22:54:26.661604 19295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "56d96ff6639f45dca8d175be5545e447"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-40vx"
server_key: "4d20552cd661188ea31b6b6c94272328"
server_key_iv: "6125d91c112eb577723943390743e904"
server_key_version: "encryptionkey@0"
I20260927 22:54:26.666566 19295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:54:26.669766 19310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:26.670991 19295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 22:54:26.671170 19295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "56d96ff6639f45dca8d175be5545e447"
format_stamp: "Formatted at 2026-09-27 22:54:26 on dist-test-slave-40vx"
server_key: "4d20552cd661188ea31b6b6c94272328"
server_key_iv: "6125d91c112eb577723943390743e904"
server_key_version: "encryptionkey@0"
I20260927 22:54:26.671412 19295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:26.721719 19295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:26.726022 19295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:26.726322 19295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:26.744544 19295 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:38587
I20260927 22:54:26.744577 19361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:38587 every 8 connection(s)
I20260927 22:54:26.745873 19295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:54:26.749507 19362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:26.750412 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 19295
I20260927 22:54:26.750653 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:54:26.751328 9010 external_mini_cluster.cc:1428] Setting key 670a7f06fc4b32a489314146be0d0902
I20260927 22:54:26.761848 19362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447: Bootstrap starting.
I20260927 22:54:26.765125 19362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:26.766047 19362 log.cc:824] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:54:26 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:26.769243 19362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447: No bootstrap required, opened a new log
I20260927 22:54:26.775774 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:26.753574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.8.204.190:38587)
0927 22:54:26.754549 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:26.754564 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:26.754964 (+ 400us) server_negotiation.cc:407] Connection header received
0927 22:54:26.757255 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:26.757281 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:26.757761 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:26.758209 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:26.759475 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:26.761372 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:26.762629 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:26.763713 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:26.763906 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:26.767850 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:26.767860 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:26.767875 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:26.767924 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:26.771704 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:26.772290 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:26.772303 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:26.772315 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:26.772504 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:26.772808 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:26.772818 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:26.772822 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:26.773484 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:26.773779 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:26.774027 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 22:54:26.774416 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":454,"threads_started":1}
I20260927 22:54:26.779201 19362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d96ff6639f45dca8d175be5545e447" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38587 } }
I20260927 22:54:26.779539 19362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:26.779635 19362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d96ff6639f45dca8d175be5545e447, State: Initialized, Role: FOLLOWER
I20260927 22:54:26.780258 19362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d96ff6639f45dca8d175be5545e447" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38587 } }
I20260927 22:54:26.780526 19362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:54:26.780673 19362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:54:26.780838 19362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:26.782788 19362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d96ff6639f45dca8d175be5545e447" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38587 } }
I20260927 22:54:26.783245 19362 leader_election.cc:302] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d96ff6639f45dca8d175be5545e447; no voters:
I20260927 22:54:26.783823 19362 leader_election.cc:288] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:54:26.783957 19367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:26.784631 19367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [term 1 LEADER]: Becoming Leader. State: Replica: 56d96ff6639f45dca8d175be5545e447, State: Running, Role: LEADER
I20260927 22:54:26.785144 19367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d96ff6639f45dca8d175be5545e447" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38587 } }
I20260927 22:54:26.785759 19362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:54:26.791258 19368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d96ff6639f45dca8d175be5545e447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d96ff6639f45dca8d175be5545e447" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38587 } } }
I20260927 22:54:26.791627 19368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:26.792104 19369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d96ff6639f45dca8d175be5545e447. Latest consensus state: current_term: 1 leader_uuid: "56d96ff6639f45dca8d175be5545e447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d96ff6639f45dca8d175be5545e447" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38587 } } }
I20260927 22:54:26.792511 19369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447 [sys.catalog]: This master's current role is: LEADER
I20260927 22:54:26.792706 19376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:54:26.802753 19376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:54:26.813195 19376 catalog_manager.cc:1347] Generated new cluster ID: 01b84307083f44309291765b9c529f04
I20260927 22:54:26.813311 19376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:54:26.838536 19376 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:54:26.840049 19376 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:54:26.846833 19376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447: Generated new TSK 0
I20260927 22:54:26.847730 19376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:54:26.922427 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38587
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:27.268209 19389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:27.268474 19389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:27.268565 19389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:27.275758 19389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:27.275900 19389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:27.276114 19389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:54:27.285511 19389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:27.286898 19389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:27.288220 19389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:27.300693 19397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.300794 19394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.300770 19395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:27.301542 19389 server_base.cc:1034] running on GCE node
I20260927 22:54:27.302135 19389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:27.303002 19389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:27.304220 19389 hybrid_clock.cc:648] HybridClock initialized: now 1790549667304156 us; error 83 us; skew 500 ppm
I20260927 22:54:27.304539 19389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:27 dist-test-slave-40vx krb5kdc[19282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:27.307586 19389 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:54:27.308964 19389 webserver.cc:460] Webserver started at http://127.8.204.129:43437/ using document root <none> and password file <none>
I20260927 22:54:27.309607 19389 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:27.309746 19389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:27.310034 19389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:27.312969 19389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "20132c1e47654a939ffed27bbbde05f5"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-40vx"
server_key: "8918360ab8f1fd8cf96d0f054d23e879"
server_key_iv: "32c9657c223378cf2cababe9bc04e3be"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.313798 19389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "20132c1e47654a939ffed27bbbde05f5"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-40vx"
server_key: "8918360ab8f1fd8cf96d0f054d23e879"
server_key_iv: "32c9657c223378cf2cababe9bc04e3be"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.318696 19389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:54:27.322220 19404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.323625 19389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:54:27.323802 19389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "20132c1e47654a939ffed27bbbde05f5"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-40vx"
server_key: "8918360ab8f1fd8cf96d0f054d23e879"
server_key_iv: "32c9657c223378cf2cababe9bc04e3be"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.324015 19389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:27.350935 19389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:27.354746 19389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:27.355032 19389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:27.355995 19389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:27.357235 19389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:27.357335 19389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.357477 19389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:27.357550 19389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.389392 19389 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:40141
I20260927 22:54:27.389431 19516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:40141 every 8 connection(s)
I20260927 22:54:27.390657 19389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:54:27.397428 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 19389
I20260927 22:54:27.397650 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:54:27.398368 9010 external_mini_cluster.cc:1428] Setting key a3321c2092dbd7a6d347252f6709c253
Sep 27 22:54:27 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:27.411612 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.393087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:60153 (local address 127.8.204.190:38587)
0927 22:54:27.393492 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:27.393496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:27.394569 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 22:54:27.396491 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:27.396496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:27.396593 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:27.396712 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:27.398287 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.399049 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.401065 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.401802 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.401960 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.405418 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:27.405427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:27.405435 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:27.405486 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:27.408534 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.409412 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.409416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.409418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.409537 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.409986 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.409988 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.409990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.410213 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:27.410314 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:27.411143 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 22:54:27.411362 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":211,"threads_started":1}
I20260927 22:54:27.413079 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.393668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:38587 (local address 127.8.204.129:60153)
0927 22:54:27.394382 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:27.394421 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:27.396241 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:27.396938 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:27.396946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:27.397427 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:27.398131 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.398149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.399231 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.399236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.400913 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.400928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.402016 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.402021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.402204 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.402866 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:27.402899 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:27.405204 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:27.408663 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.408675 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.408693 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.409299 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.409633 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.409641 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.409647 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.409871 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.410327 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:27.410337 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:27.410750 (+ 413us) client_negotiation.cc:769] Sending connection context
0927 22:54:27.411053 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 22:54:27.411624 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":263,"threads_started":1}
I20260927 22:54:27.414621 19517 heartbeater.cc:347] Connected to a master server at 127.8.204.190:38587
I20260927 22:54:27.414906 19517 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:27.415643 19517 heartbeater.cc:511] Master 127.8.204.190:38587 requested a full tablet report, sending...
I20260927 22:54:27.417564 19327 ts_manager.cc:194] Registered new tserver with Master: 20132c1e47654a939ffed27bbbde05f5 (127.8.204.129:40141)
I20260927 22:54:27.419000 19327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:60153
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:54:27.450266 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38587
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:27.772447 19523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:27.772763 19523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:27.772924 19523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:27.780009 19523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:27.780150 19523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:27.780314 19523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:54:27.789664 19523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:27.790906 19523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:27.792187 19523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:27.804646 19528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.804680 19529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:27.805151 19531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:27.805627 19523 server_base.cc:1034] running on GCE node
I20260927 22:54:27.806385 19523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:27.807369 19523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:27.808621 19523 hybrid_clock.cc:648] HybridClock initialized: now 1790549667808539 us; error 53 us; skew 500 ppm
I20260927 22:54:27.808925 19523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:27 dist-test-slave-40vx krb5kdc[19282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:27.813756 19523 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:54:27.815666 19523 webserver.cc:460] Webserver started at http://127.8.204.130:36353/ using document root <none> and password file <none>
I20260927 22:54:27.816354 19523 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:27.816500 19523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:27.816838 19523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:27.819766 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "ca12d491753d4232807884f4a962f77b"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-40vx"
server_key: "a9f0019c18ab8cefdcbf222a637bde2b"
server_key_iv: "c88d54aaedb56752fbb4e4d07f8a8776"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.820550 19523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "ca12d491753d4232807884f4a962f77b"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-40vx"
server_key: "a9f0019c18ab8cefdcbf222a637bde2b"
server_key_iv: "c88d54aaedb56752fbb4e4d07f8a8776"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.825814 19523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:54:27.829319 19538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.830593 19523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:54:27.830765 19523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "ca12d491753d4232807884f4a962f77b"
format_stamp: "Formatted at 2026-09-27 22:54:27 on dist-test-slave-40vx"
server_key: "a9f0019c18ab8cefdcbf222a637bde2b"
server_key_iv: "c88d54aaedb56752fbb4e4d07f8a8776"
server_key_version: "encryptionkey@0"
I20260927 22:54:27.830967 19523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:27.844565 19523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:27.849016 19523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:27.849290 19523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:27.850194 19523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:27.851488 19523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:27.851586 19523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.851723 19523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:27.851797 19523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:27.883335 19523 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38741
I20260927 22:54:27.883396 19650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38741 every 8 connection(s)
I20260927 22:54:27.884685 19523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:54:27.889290 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 19523
I20260927 22:54:27.889523 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:54:27.889941 9010 external_mini_cluster.cc:1428] Setting key 83da2bb63281a6c5f69508004951f401
Sep 27 22:54:27 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549667, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:27.906044 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.887385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:41639 (local address 127.8.204.190:38587)
0927 22:54:27.887560 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:27.887564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:27.888694 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 22:54:27.890672 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:27.890678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:27.890774 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:27.890887 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:27.892752 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.893949 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.895626 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.896391 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.896526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.900091 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:27.900099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:27.900107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:27.900152 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:27.902802 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.903767 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.903771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.903773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.903921 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:27.904397 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:27.904400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:27.904402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:27.904635 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:27.904738 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:27.905628 (+ 890us) server_negotiation.cc:299] Negotiation successful
0927 22:54:27.905791 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:54:27.907686 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:27.887806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:38587 (local address 127.8.204.130:41639)
0927 22:54:27.888534 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:27.888571 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:27.890327 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:27.891156 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:27.891166 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:27.891957 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:27.892590 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.892607 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.894099 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.894102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.895495 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:27.895507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:27.896591 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:27.896596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:27.896790 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:27.897467 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:27.897487 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:27.899834 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:27.902941 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.902952 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.902970 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.903636 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.904032 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:27.904039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:27.904046 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:27.904280 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:27.904760 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:27.904770 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:27.905213 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 22:54:27.905507 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 22:54:27.906061 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260927 22:54:27.909118 19651 heartbeater.cc:347] Connected to a master server at 127.8.204.190:38587
I20260927 22:54:27.909428 19651 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:27.910223 19651 heartbeater.cc:511] Master 127.8.204.190:38587 requested a full tablet report, sending...
I20260927 22:54:27.911711 19327 ts_manager.cc:194] Registered new tserver with Master: ca12d491753d4232807884f4a962f77b (127.8.204.130:38741)
I20260927 22:54:27.912523 19327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:41639
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:54:27.943424 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38587
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:54:28.260942 19656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:54:28.261188 19656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:54:28.261287 19656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:54:28.268218 19656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:54:28.268313 19656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:54:28.268424 19656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:54:28.277581 19656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:54:28.278698 19656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:54:28.279918 19656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:54:28.292415 19661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:28.292502 19664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:54:28.292532 19662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:54:28.293830 19656 server_base.cc:1034] running on GCE node
I20260927 22:54:28.294463 19656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:54:28.295509 19656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:54:28.296725 19656 hybrid_clock.cc:648] HybridClock initialized: now 1790549668296685 us; error 27 us; skew 500 ppm
I20260927 22:54:28.297062 19656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:54:28 dist-test-slave-40vx krb5kdc[19282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:54:28.300220 19656 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:54:28.301606 19656 webserver.cc:460] Webserver started at http://127.8.204.131:34647/ using document root <none> and password file <none>
I20260927 22:54:28.302202 19656 fs_manager.cc:359] Metadata directory not provided
I20260927 22:54:28.302299 19656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:54:28.302538 19656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:54:28.305299 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "8facf590c3c74b37a8f9f0b92540ed24"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-40vx"
server_key: "4a8c38306cdf454abfb4c826b3e35b17"
server_key_iv: "2e74237f4754271f28fb4186edefb085"
server_key_version: "encryptionkey@0"
I20260927 22:54:28.306005 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "8facf590c3c74b37a8f9f0b92540ed24"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-40vx"
server_key: "4a8c38306cdf454abfb4c826b3e35b17"
server_key_iv: "2e74237f4754271f28fb4186edefb085"
server_key_version: "encryptionkey@0"
I20260927 22:54:28.310739 19656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 22:54:28.313898 19671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:28.315131 19656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 22:54:28.315358 19656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "8facf590c3c74b37a8f9f0b92540ed24"
format_stamp: "Formatted at 2026-09-27 22:54:28 on dist-test-slave-40vx"
server_key: "4a8c38306cdf454abfb4c826b3e35b17"
server_key_iv: "2e74237f4754271f28fb4186edefb085"
server_key_version: "encryptionkey@0"
I20260927 22:54:28.315553 19656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:54:28.341356 19656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:54:28.345515 19656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:54:28.345804 19656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:54:28.346705 19656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:54:28.348014 19656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:54:28.348114 19656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:28.348260 19656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:54:28.348332 19656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:54:28.380810 19656 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:35539
I20260927 22:54:28.380896 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:35539 every 8 connection(s)
I20260927 22:54:28.382040 19656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:54:28.384876 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 19656
I20260927 22:54:28.385064 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:54:28.385535 9010 external_mini_cluster.cc:1428] Setting key 60a6121a46f56f60959ee20c99c9713d
Sep 27 22:54:28 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549668, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:54:28.403610 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.384548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:54349 (local address 127.8.204.190:38587)
0927 22:54:28.384763 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.384769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.386072 (+ 1303us) server_negotiation.cc:407] Connection header received
0927 22:54:28.388195 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.388200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.388315 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.388420 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.390336 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.391074 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.393366 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.393787 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.393911 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.397581 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.397585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.397589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.397621 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.400724 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.401516 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.401520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.401522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.401658 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.402026 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.402029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.402031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.402289 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.402398 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.403046 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.403221 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:54:28.404495 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.385072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:38587 (local address 127.8.204.131:54349)
0927 22:54:28.385886 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.385930 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.387923 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.388667 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.388678 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.389357 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:54:28.390137 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.390165 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.391221 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.391226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.393198 (+ 1972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.393218 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.394073 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.394081 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.394362 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.395092 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:54:28.395124 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:54:28.397384 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:54:28.400890 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:28.400899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:28.400914 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:28.401366 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:28.401755 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:54:28.401758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:54:28.401760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:54:28.401931 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:54:28.402391 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:54:28.402396 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:54:28.402699 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.402913 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.403412 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":321,"threads_started":1}
I20260927 22:54:28.405956 19784 heartbeater.cc:347] Connected to a master server at 127.8.204.190:38587
I20260927 22:54:28.406246 19784 heartbeater.cc:464] Registering TS with master...
I20260927 22:54:28.406952 19784 heartbeater.cc:511] Master 127.8.204.190:38587 requested a full tablet report, sending...
I20260927 22:54:28.408371 19327 ts_manager.cc:194] Registered new tserver with Master: 8facf590c3c74b37a8f9f0b92540ed24 (127.8.204.131:35539)
I20260927 22:54:28.409160 19327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:54349
I20260927 22:54:28.413391 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:54:28.421783 19517 heartbeater.cc:503] Master 127.8.204.190:38587 was elected leader, sending a full tablet report...
I20260927 22:54:28.430058 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.417654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.8.204.190:38587)
0927 22:54:28.417886 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.417893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.418263 (+ 370us) server_negotiation.cc:407] Connection header received
0927 22:54:28.418460 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.418466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.418534 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.418633 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.420052 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.420827 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.422109 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.422623 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.422759 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.425059 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.425065 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.425070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.425109 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.427443 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.428092 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.428100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.428104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.428249 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.428607 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.428613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.428617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.428980 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.429221 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.429539 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.429734 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:54:28.433480 19327 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:54:28.440044 19327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:54:28.461052 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.451112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.8.204.129:40141)
0927 22:54:28.451910 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.451922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.452380 (+ 458us) server_negotiation.cc:407] Connection header received
0927 22:54:28.452760 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.452768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.453035 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.453189 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.454085 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.455727 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.458395 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.459900 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.460025 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.460245 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.460567 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.460730 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":379,"threads_started":1}
I20260927 22:54:28.461483 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.451431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:35539 (local address 127.0.0.1:53276)
0927 22:54:28.452334 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.452359 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.452530 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.454446 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.454452 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.454474 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.454868 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.454878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.457033 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.457042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.458952 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.458968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.460696 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.460701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.460853 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.460885 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.460944 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.461070 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":87,"spinlock_wait_cycles":8704,"thread_start_us":663,"threads_started":1}
I20260927 22:54:28.461179 19798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.450967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40141 (local address 127.0.0.1:53830)
0927 22:54:28.452240 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.452283 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.452530 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.453425 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.453434 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.453487 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.453894 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.453906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.455878 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.455883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.458220 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.458235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.460152 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.460158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.460330 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.460374 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.460513 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.460703 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":1,"spinlock_wait_cycles":2944,"thread_start_us":1108,"threads_started":1}
I20260927 22:54:28.462013 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.451974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.8.204.131:35539)
0927 22:54:28.453862 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.453872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.453903 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:54:28.454005 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.454012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.454231 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.455058 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.455336 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.456893 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.459114 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.460498 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.461273 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.461487 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.461551 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.461700 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":1559,"threads_started":1}
I20260927 22:54:28.461863 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.452907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38741 (local address 127.0.0.1:39948)
0927 22:54:28.454528 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.454544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.454687 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.455194 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.455198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.455220 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.455557 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.455565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.457127 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.457136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.459018 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.459055 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.461077 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.461085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.461260 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.461286 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.461386 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.461512 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1508,"mutex_wait_us":55,"thread_start_us":1364,"threads_started":1}
I20260927 22:54:28.463038 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.453424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.8.204.130:38741)
0927 22:54:28.454102 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.454114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.454647 (+ 533us) server_negotiation.cc:407] Connection header received
0927 22:54:28.454856 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.454865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.455019 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.455188 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.455690 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.456988 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.459359 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.460621 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.460744 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.460917 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.461955 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.462846 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":423,"threads_started":1}
I20260927 22:54:28.465772 19586 tablet_service.cc:1467] Processing CreateTablet for tablet 62b7f88aa10c40d1b3c8b27cd9387a98 (DEFAULT_TABLE table=test-table [id=d9e8080b7e9c4ed89eb57b98a96929b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:28.465674 19719 tablet_service.cc:1467] Processing CreateTablet for tablet 62b7f88aa10c40d1b3c8b27cd9387a98 (DEFAULT_TABLE table=test-table [id=d9e8080b7e9c4ed89eb57b98a96929b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:28.465673 19452 tablet_service.cc:1467] Processing CreateTablet for tablet 62b7f88aa10c40d1b3c8b27cd9387a98 (DEFAULT_TABLE table=test-table [id=d9e8080b7e9c4ed89eb57b98a96929b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:54:28.466905 19586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b7f88aa10c40d1b3c8b27cd9387a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:28.467013 19452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b7f88aa10c40d1b3c8b27cd9387a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:28.467026 19719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b7f88aa10c40d1b3c8b27cd9387a98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:54:28.478097 19802 tablet_bootstrap.cc:492] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Bootstrap starting.
I20260927 22:54:28.480913 19803 tablet_bootstrap.cc:492] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Bootstrap starting.
I20260927 22:54:28.481675 19804 tablet_bootstrap.cc:492] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Bootstrap starting.
I20260927 22:54:28.482769 19802 tablet_bootstrap.cc:654] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:28.484103 19802 log.cc:824] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:28.484428 19804 tablet_bootstrap.cc:654] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:28.484990 19803 tablet_bootstrap.cc:654] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Neither blocks nor log segments found. Creating new log.
I20260927 22:54:28.485606 19804 log.cc:824] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:28.486171 19803 log.cc:824] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:54:28.487205 19802 tablet_bootstrap.cc:492] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: No bootstrap required, opened a new log
I20260927 22:54:28.487501 19802 ts_tablet_manager.cc:1397] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260927 22:54:28.489059 19803 tablet_bootstrap.cc:492] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: No bootstrap required, opened a new log
I20260927 22:54:28.489053 19804 tablet_bootstrap.cc:492] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: No bootstrap required, opened a new log
I20260927 22:54:28.489295 19804 ts_tablet_manager.cc:1397] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:54:28.489295 19803 ts_tablet_manager.cc:1397] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 22:54:28.494520 19802 raft_consensus.cc:348] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.494868 19802 raft_consensus.cc:374] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:28.495002 19802 raft_consensus.cc:729] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca12d491753d4232807884f4a962f77b, State: Initialized, Role: FOLLOWER
I20260927 22:54:28.495621 19802 consensus_queue.cc:260] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.496675 19651 heartbeater.cc:503] Master 127.8.204.190:38587 was elected leader, sending a full tablet report...
I20260927 22:54:28.497287 19802 ts_tablet_manager.cc:1428] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:54:28.499790 19803 raft_consensus.cc:348] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.500176 19803 raft_consensus.cc:374] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:28.500257 19803 raft_consensus.cc:729] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20132c1e47654a939ffed27bbbde05f5, State: Initialized, Role: FOLLOWER
I20260927 22:54:28.500901 19803 consensus_queue.cc:260] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.500912 19804 raft_consensus.cc:348] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.501276 19804 raft_consensus.cc:374] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:54:28.501386 19804 raft_consensus.cc:729] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8facf590c3c74b37a8f9f0b92540ed24, State: Initialized, Role: FOLLOWER
I20260927 22:54:28.502046 19804 consensus_queue.cc:260] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.502581 19803 ts_tablet_manager.cc:1428] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260927 22:54:28.503959 19784 heartbeater.cc:503] Master 127.8.204.190:38587 was elected leader, sending a full tablet report...
I20260927 22:54:28.504521 19804 ts_tablet_manager.cc:1428] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260927 22:54:28.524359 19810 raft_consensus.cc:482] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:54:28.524601 19810 raft_consensus.cc:504] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.526027 19810 leader_election.cc:288] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20132c1e47654a939ffed27bbbde05f5 (127.8.204.129:40141), ca12d491753d4232807884f4a962f77b (127.8.204.130:38741)
I20260927 22:54:28.531987 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.526379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40141 (local address 127.8.204.131:49517)
0927 22:54:28.526589 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.526605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.526726 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.527191 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.527197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.527221 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.527445 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.527453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.528543 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.528549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.530243 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.530254 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.531462 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.531466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.531584 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.531607 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.531691 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.531772 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":41}
I20260927 22:54:28.532111 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.526472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:49517 (local address 127.8.204.129:40141)
0927 22:54:28.526682 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.526689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.526708 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:54:28.526885 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.526896 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.526991 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.527079 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.527578 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.528405 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.530396 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.531335 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.531498 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.531568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.531741 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.531880 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:54:28.533133 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.526896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:33101 (local address 127.8.204.130:38741)
0927 22:54:28.527103 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.527109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.527588 (+ 479us) server_negotiation.cc:407] Connection header received
0927 22:54:28.527801 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.527806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.527866 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.527951 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:54:28.528397 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.529378 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.531030 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.532304 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.532470 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.532562 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.532866 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.532953 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:54:28.533051 19472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b7f88aa10c40d1b3c8b27cd9387a98" candidate_uuid: "8facf590c3c74b37a8f9f0b92540ed24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20132c1e47654a939ffed27bbbde05f5" is_pre_election: true
I20260927 22:54:28.533342 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.526753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38741 (local address 127.8.204.131:33101)
0927 22:54:28.527496 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 22:54:28.527509 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:54:28.527681 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:54:28.527973 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:54:28.527977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:54:28.528000 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:54:28.528268 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.528275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.529549 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.529553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.530885 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:54:28.530895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.532520 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.532525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:54:28.532705 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.532741 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:54:28.532838 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:54:28.532961 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":22,"thread_start_us":438,"threads_started":1}
I20260927 22:54:28.533634 19472 raft_consensus.cc:2454] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8facf590c3c74b37a8f9f0b92540ed24 in term 0.
I20260927 22:54:28.534305 19673 leader_election.cc:302] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20132c1e47654a939ffed27bbbde05f5, 8facf590c3c74b37a8f9f0b92540ed24; no voters:
I20260927 22:54:28.534265 19606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b7f88aa10c40d1b3c8b27cd9387a98" candidate_uuid: "8facf590c3c74b37a8f9f0b92540ed24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca12d491753d4232807884f4a962f77b" is_pre_election: true
I20260927 22:54:28.534693 19810 raft_consensus.cc:2781] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:54:28.534835 19810 raft_consensus.cc:482] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:54:28.534791 19606 raft_consensus.cc:2454] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8facf590c3c74b37a8f9f0b92540ed24 in term 0.
I20260927 22:54:28.534950 19810 raft_consensus.cc:3037] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:28.537204 19810 raft_consensus.cc:504] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.537735 19810 leader_election.cc:288] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [CANDIDATE]: Term 1 election: Requested vote from peers 20132c1e47654a939ffed27bbbde05f5 (127.8.204.129:40141), ca12d491753d4232807884f4a962f77b (127.8.204.130:38741)
I20260927 22:54:28.538295 19472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b7f88aa10c40d1b3c8b27cd9387a98" candidate_uuid: "8facf590c3c74b37a8f9f0b92540ed24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20132c1e47654a939ffed27bbbde05f5"
I20260927 22:54:28.538385 19606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62b7f88aa10c40d1b3c8b27cd9387a98" candidate_uuid: "8facf590c3c74b37a8f9f0b92540ed24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca12d491753d4232807884f4a962f77b"
I20260927 22:54:28.538530 19472 raft_consensus.cc:3037] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:28.538620 19606 raft_consensus.cc:3037] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:54:28.540757 19472 raft_consensus.cc:2454] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8facf590c3c74b37a8f9f0b92540ed24 in term 1.
I20260927 22:54:28.541078 19673 leader_election.cc:302] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20132c1e47654a939ffed27bbbde05f5, 8facf590c3c74b37a8f9f0b92540ed24; no voters:
I20260927 22:54:28.541345 19606 raft_consensus.cc:2454] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8facf590c3c74b37a8f9f0b92540ed24 in term 1.
I20260927 22:54:28.541364 19810 raft_consensus.cc:2781] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:54:28.541906 19810 raft_consensus.cc:686] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 1 LEADER]: Becoming Leader. State: Replica: 8facf590c3c74b37a8f9f0b92540ed24, State: Running, Role: LEADER
I20260927 22:54:28.542609 19810 consensus_queue.cc:237] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } }
I20260927 22:54:28.546779 19326 catalog_manager.cc:5495] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8facf590c3c74b37a8f9f0b92540ed24 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "8facf590c3c74b37a8f9f0b92540ed24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8facf590c3c74b37a8f9f0b92540ed24" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 35539 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 } health_report { overall_health: UNKNOWN } } }
I20260927 22:54:28.584803 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.579372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.8.204.131:35539)
0927 22:54:28.579630 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.579637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.579747 (+ 110us) server_negotiation.cc:407] Connection header received
0927 22:54:28.579998 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.580008 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.580091 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.580253 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:54:28.580619 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.581369 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.582706 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.583102 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.583237 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.583736 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:54:28.584229 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:54:28.584386 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.584462 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.584539 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:54:28.592551 19606 raft_consensus.cc:1261] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 1 FOLLOWER]: Refusing update from remote peer 8facf590c3c74b37a8f9f0b92540ed24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:28.592607 19472 raft_consensus.cc:1261] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 1 FOLLOWER]: Refusing update from remote peer 8facf590c3c74b37a8f9f0b92540ed24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:54:28.593478 19812 consensus_queue.cc:1035] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20132c1e47654a939ffed27bbbde05f5" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:28.593889 19810 consensus_queue.cc:1035] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca12d491753d4232807884f4a962f77b" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:54:28.620671 19326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:28.620959 19326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:54:28.624397 19326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d96ff6639f45dca8d175be5545e447: Sending DeleteTablet for 3 replicas of tablet 62b7f88aa10c40d1b3c8b27cd9387a98
I20260927 22:54:28.625420 19452 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b7f88aa10c40d1b3c8b27cd9387a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:28 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:53830
I20260927 22:54:28.625476 19719 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b7f88aa10c40d1b3c8b27cd9387a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:28 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:53276
I20260927 22:54:28.625764 19586 tablet_service.cc:1514] Processing DeleteTablet for tablet 62b7f88aa10c40d1b3c8b27cd9387a98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:54:28 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:39948
I20260927 22:54:28.627975 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 19389
I20260927 22:54:28.645411 19827 tablet_replica.cc:330] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: stopping tablet replica
I20260927 22:54:28.646018 19827 raft_consensus.cc:2229] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:28.646678 19827 raft_consensus.cc:2258] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:28.647533 19826 tablet_replica.cc:330] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: stopping tablet replica
I20260927 22:54:28.648017 19825 tablet_replica.cc:330] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: stopping tablet replica
I20260927 22:54:28.648015 19826 raft_consensus.cc:2229] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:54:28.648425 19825 raft_consensus.cc:2229] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:54:28.648598 19826 raft_consensus.cc:2258] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:28.648916 19825 raft_consensus.cc:2258] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:54:28.650420 19827 ts_tablet_manager.cc:1905] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:28.650910 19825 ts_tablet_manager.cc:1905] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:54:28 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:54:28.652843 19826 ts_tablet_manager.cc:1905] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:54:28.657994 19825 ts_tablet_manager.cc:1918] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:28.658159 19825 log.cc:1196] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/62b7f88aa10c40d1b3c8b27cd9387a98
I20260927 22:54:28.658674 19825 ts_tablet_manager.cc:1939] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 20132c1e47654a939ffed27bbbde05f5: Deleting consensus metadata
I20260927 22:54:28.660492 19312 catalog_manager.cc:4841] TS 20132c1e47654a939ffed27bbbde05f5 (127.8.204.129:40141): tablet 62b7f88aa10c40d1b3c8b27cd9387a98 (table test-table [id=d9e8080b7e9c4ed89eb57b98a96929b0]) successfully deleted
I20260927 22:54:28.661132 19797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.629353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.8.204.129:40141)
0927 22:54:28.629574 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.629583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.635285 (+ 5702us) server_negotiation.cc:407] Connection header received
0927 22:54:28.636151 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.636160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.636264 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.636353 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.640358 (+ 4005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.641822 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.645172 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.646127 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.646248 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.653383 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.653396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.653412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.653455 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.657340 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.658155 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.658164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.658171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.658358 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.658844 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.658850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.658853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.659318 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.659619 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.660147 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.660887 (+ 740us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:54:28.662273 19512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:28.665373 19827 ts_tablet_manager.cc:1918] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:28.665537 19827 log.cc:1196] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/62b7f88aa10c40d1b3c8b27cd9387a98
I20260927 22:54:28.666060 19827 ts_tablet_manager.cc:1939] T 62b7f88aa10c40d1b3c8b27cd9387a98 P ca12d491753d4232807884f4a962f77b: Deleting consensus metadata
I20260927 22:54:28.667826 19312 catalog_manager.cc:4841] TS ca12d491753d4232807884f4a962f77b (127.8.204.130:38741): tablet 62b7f88aa10c40d1b3c8b27cd9387a98 (table test-table [id=d9e8080b7e9c4ed89eb57b98a96929b0]) successfully deleted
I20260927 22:54:28.668079 19826 ts_tablet_manager.cc:1918] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:54:28.668254 19826 log.cc:1196] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/62b7f88aa10c40d1b3c8b27cd9387a98
I20260927 22:54:28.668814 19826 ts_tablet_manager.cc:1939] T 62b7f88aa10c40d1b3c8b27cd9387a98 P 8facf590c3c74b37a8f9f0b92540ed24: Deleting consensus metadata
I20260927 22:54:28.670379 19313 catalog_manager.cc:4841] TS 8facf590c3c74b37a8f9f0b92540ed24 (127.8.204.131:35539): tablet 62b7f88aa10c40d1b3c8b27cd9387a98 (table test-table [id=d9e8080b7e9c4ed89eb57b98a96929b0]) successfully deleted
I20260927 22:54:28.752655 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 19389
I20260927 22:54:28.770084 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 19523
Sep 27 22:54:28 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:54:28.784785 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.770911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.8.204.130:38741)
0927 22:54:28.771078 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.771086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.771128 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:54:28.771361 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.771369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.771464 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.771581 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.772716 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.773429 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.774645 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.775155 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.775300 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.779443 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.779456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.779477 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.779506 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.782176 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.782798 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.782805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.782808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.782912 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.783266 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.783293 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.783299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.783695 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.783975 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.784260 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.784462 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:54:28.785451 19646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:28.898154 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 19523
I20260927 22:54:28.915839 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 19656
Sep 27 22:54:28 dist-test-slave-40vx krb5kdc[19282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:54:28.933560 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:54:28.916684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.8.204.131:35539)
0927 22:54:28.916963 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 22:54:28.916972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:54:28.916999 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:54:28.917055 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:54:28.917059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:54:28.917122 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:54:28.917205 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:54:28.918946 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.919947 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.921726 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:54:28.922189 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:54:28.922283 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:54:28.927695 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:54:28.927705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:54:28.927719 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:54:28.927759 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:54:28.930707 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.931442 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.931457 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.931463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.931659 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:54:28.932048 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:54:28.932053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:54:28.932056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:54:28.932380 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:54:28.932705 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:54:28.932995 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:54:28.933222 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:54:28.934198 19779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:29.055953 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 19656
I20260927 22:54:29.076133 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 19295
I20260927 22:54:29.077304 19357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:54:29.182204 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 19295
2026-09-27T22:54:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4201 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:54:29 dist-test-slave-40vx krb5kdc[19835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:54:29 dist-test-slave-40vx krb5kdc[19835](info): set up 2 sockets
Sep 27 22:54:29 dist-test-slave-40vx krb5kdc[19835](info): commencing operation
krb5kdc: starting...
W20260927 22:54:30.280104 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:54:30 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:54:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:54:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/rangeradmin_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/rangeradmin_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/rangerlookup_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/rangerlookup_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
I20260927 22:54:30.594911 9010 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskoq4u9H/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/postgres -l logfile start
2026-09-27 22:54:33.096 UTC [19865] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:54:33.096 UTC [19865] LOG: listening on IPv4 address "127.8.204.148", port 45771
2026-09-27 22:54:33.099 UTC [19865] LOG: listening on Unix socket "/tmp/.s.PGSQL.45771"
2026-09-27 22:54:33.121 UTC [19868] LOG: database system was shut down at 2026-09-27 22:54:31 UTC
2026-09-27 22:54:33.124 UTC [19865] LOG: database system is ready to accept connections
I20260927 22:54:34.099830 9010 mini_postgres.cc:96] Postgres bound to 45771
2026-09-27 22:54:34.116 UTC [19876] FATAL: database "slave" does not exist
127.8.204.148:45771 - accepting connections
I20260927 22:54:34.117170 9010 mini_ranger.cc:161] Starting Ranger...
I20260927 22:54:34.140627 9010 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:54:34.636325 9010 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:54:34.636462 9010 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-admin
2026-09-27 22:54:34,925 [I] DB FLAVOR :POSTGRES
2026-09-27 22:54:34,925 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:54:34,925 [I] Checking connection..
2026-09-27 22:54:34,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:54:35,175 [I] Checking connection passed.
2026-09-27 22:54:35,176 [I] --------- Verifying version history table ---------
2026-09-27 22:54:35,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:54:35,424 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:54:35,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:54:35,687 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:54:35,687 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:54:35,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:54:35.935 UTC [19940] WARNING: there is no transaction in progress
2026-09-27 22:54:35,941 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:54:35,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:54:36,195 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:54:36,196 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:54:36,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:54:36,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:54:36,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-40vx.c.gcp-upstream.internal','N') ;"
2026-09-27 22:54:36,899 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:54:37,126 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:54:37,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:54:37,383 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:54:37,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:54:37,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:54:37,874 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:54:37,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:54:38.808 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.824 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.841 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.855 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.871 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.888 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.942 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.949 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.962 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.973 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.983 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:38.997 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:39.007 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:39.013 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:39.020 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:39.027 UTC [20048] WARNING: there is no transaction in progress
2026-09-27 22:54:39,644 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:54:39,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:54:39,889 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:54:39,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:54:40,117 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:54:40,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:54:40,351 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:54:40,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:54:40,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:54:40,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-40vx.c.gcp-upstream.internal';"
2026-09-27 22:54:41,099 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:54:41,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:54:41,331 [I] DB_PATCHES have already been applied
I20260927 22:54:41.336796 9010 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-admin:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskoq4u9H/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:54:41.336889 9010 mini_ranger.cc:193] Using host: 127.8.204.148
I20260927 22:54:41.351344 9010 mini_ranger.cc:227] Ranger admin URL: http://127.8.204.148:56231
Sep 27, 2026 10:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:54:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:54:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:54:42.495024 9010 mini_ranger.cc:160] Time spent starting Ranger: real 8.378s user 0.005s sys 0.030s
Sep 27, 2026 10:55:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:55:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:55:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:55:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:55:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:55:03.192122 9010 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/rangerkms_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/rangerkms_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.204.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:55:03 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549703, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:55:03.368299 9010 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:55:03.393266 9010 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:55:04.298146 9010 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:55:04.298323 9010 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms
I20260927 22:55:04.298439 9010 mini_ranger_kms.cc:227] Using postgres at 127.8.204.148:45771
2026-09-27 22:55:04,710 [I] DB FLAVOR :POSTGRES
2026-09-27 22:55:04,710 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:55:04,710 [I] Checking connection
2026-09-27 22:55:04,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:55:04,936 [I] connection success
2026-09-27 22:55:04,936 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:55:04,937 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:55:04,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:55:05,166 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:55:05,167 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:55:05,167 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:55:05,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:45771/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:55:05,418 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:55:05.852823 9010 mini_ranger_kms.cc:310] Created kms service
I20260927 22:55:06.052623 9010 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:55:06.095163 9010 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:55:06 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549703, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:55:06.446343 9010 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:55:06.446660 9010 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms:/tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:55:06.446729 9010 mini_ranger_kms.cc:240] Using host: 127.8.204.148
I20260927 22:55:06.467583 9010 mini_ranger_kms.cc:276] Ranger KMS PID: 20228
I20260927 22:55:06.467775 9010 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.204.148:42303
Sep 27, 2026 10:55:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:55:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:55:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:55:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:55:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:55:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:55:08.759954 9010 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.392s user 0.131s sys 0.300s
I20260927 22:55:08.760210 9010 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:55:08.760412 9010 mini_ranger_kms.cc:401] 127.8.204.148:42303/kms/v1/keys
Sep 27 22:55:08 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549708, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:55:09 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549708, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.148@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
Sep 27 22:55:09 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549708, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.148@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
Sep 27 22:55:11 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:55:11.823462 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:38851
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:38851
--ranger_config_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:42303/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:55:12.160580 20279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:12.160862 20279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:12.161039 20279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:12.168223 20279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:55:12.168772 20279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:12.168875 20279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:12.168982 20279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:55:12.169097 20279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:55:12.179225 20279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.204.148:42303/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:38851
--ranger_config_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:38851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:55:12.180594 20279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:12.181910 20279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:12.193945 20284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:12.193985 20287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:12.194020 20285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:12.194734 20279 server_base.cc:1034] running on GCE node
I20260927 22:55:12.195545 20279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:12.197000 20279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:12.198220 20279 hybrid_clock.cc:648] HybridClock initialized: now 1790549712198188 us; error 48 us; skew 500 ppm
I20260927 22:55:12.198563 20279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:12 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549712, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:12.201599 20279 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:55:12.203142 20279 webserver.cc:460] Webserver started at http://127.8.204.190:34673/ using document root <none> and password file <none>
I20260927 22:55:12.203833 20279 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:12.203965 20279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:12.204264 20279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:55:12 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549712, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:55:12.253973 20279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "cc11083b97d640918d6b4fd57105a7e7"
format_stamp: "Formatted at 2026-09-27 22:55:12 on dist-test-slave-40vx"
server_key: "dec7ed2cf2d96f479f75994d7eb99922"
server_key_iv: "3d0f862c5b92b9101ddc8aa10072765a"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:12.255066 20279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "cc11083b97d640918d6b4fd57105a7e7"
format_stamp: "Formatted at 2026-09-27 22:55:12 on dist-test-slave-40vx"
server_key: "dec7ed2cf2d96f479f75994d7eb99922"
server_key_iv: "3d0f862c5b92b9101ddc8aa10072765a"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:12.262288 20279 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:55:12.279686 20294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:12.281499 20279 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:55:12.281733 20279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "cc11083b97d640918d6b4fd57105a7e7"
format_stamp: "Formatted at 2026-09-27 22:55:12 on dist-test-slave-40vx"
server_key: "dec7ed2cf2d96f479f75994d7eb99922"
server_key_iv: "3d0f862c5b92b9101ddc8aa10072765a"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:12.281984 20279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:12.306609 20279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:12.310819 20279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:12.311147 20279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:12.331825 20279 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:38851
I20260927 22:55:12.331911 20345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:38851 every 8 connection(s)
I20260927 22:55:12.333206 20279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:55:12.337697 20346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:12.339684 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 20279
I20260927 22:55:12.339900 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:55:12.340528 9010 external_mini_cluster.cc:1428] Setting key f4edc706d8f3456db55fb3675493b308
I20260927 22:55:12.352321 20346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7: Bootstrap starting.
I20260927 22:55:12.356256 20346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:12.357676 20346 log.cc:824] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:12.361342 20346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7: No bootstrap required, opened a new log
Sep 27 22:55:12 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:55:12.372370 20346 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc11083b97d640918d6b4fd57105a7e7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38851 } }
I20260927 22:55:12.372901 20346 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:12.373060 20346 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc11083b97d640918d6b4fd57105a7e7, State: Initialized, Role: FOLLOWER
I20260927 22:55:12.373787 20346 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc11083b97d640918d6b4fd57105a7e7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38851 } }
I20260927 22:55:12.374064 20346 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:55:12.374228 20346 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:55:12.374408 20346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:12.378213 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:12.342739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.8.204.190:38851)
0927 22:55:12.357596 (+ 14857us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:12.357611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:12.357663 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:55:12.359996 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:12.360018 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:12.360397 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:12.360782 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:12.362156 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:12.363509 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:12.364795 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:12.365958 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:12.366186 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:12.369978 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:12.369987 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:12.370003 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:12.370041 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:12.373395 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:12.374234 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:12.374242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:12.374249 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:12.374464 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:12.374869 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:12.374875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:12.374878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:12.375544 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:12.375883 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:12.376329 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 22:55:12.376805 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14614,"thread_start_us":14355,"threads_started":1}
I20260927 22:55:12.378120 20346 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc11083b97d640918d6b4fd57105a7e7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38851 } }
I20260927 22:55:12.378801 20346 leader_election.cc:302] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc11083b97d640918d6b4fd57105a7e7; no voters:
I20260927 22:55:12.379485 20346 leader_election.cc:288] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:55:12.379609 20351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:12.383960 20351 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [term 1 LEADER]: Becoming Leader. State: Replica: cc11083b97d640918d6b4fd57105a7e7, State: Running, Role: LEADER
I20260927 22:55:12.384572 20351 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc11083b97d640918d6b4fd57105a7e7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38851 } }
I20260927 22:55:12.384804 20346 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:55:12.387217 20352 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc11083b97d640918d6b4fd57105a7e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc11083b97d640918d6b4fd57105a7e7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38851 } } }
I20260927 22:55:12.387622 20352 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:12.388538 20353 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc11083b97d640918d6b4fd57105a7e7. Latest consensus state: current_term: 1 leader_uuid: "cc11083b97d640918d6b4fd57105a7e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc11083b97d640918d6b4fd57105a7e7" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 38851 } } }
I20260927 22:55:12.388813 20353 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:55:12.402410 20356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:55:12.411216 20356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:55:12.429608 20346 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:55:12.430231 20356 catalog_manager.cc:1347] Generated new cluster ID: 86fda30345c6427493c711bce139ede8
I20260927 22:55:12.430399 20356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:55:12.456449 20356 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:55:12.458636 20356 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:55:12.471341 20356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7: Generated new TSK 0
I20260927 22:55:12.472656 20356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:55:13.325626 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:42303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38851
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:55:13 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:55:13.752238 20391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:13.752629 20391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:13.752772 20391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:13.764635 20391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:13.764784 20391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:13.765040 20391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:55:13.781383 20391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.204.148:42303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:55:13.783403 20391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:13.785332 20391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:13.805568 20402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:13.812328 20401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:13.819965 20391 server_base.cc:1034] running on GCE node
W20260927 22:55:13.820286 20404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:13.821193 20391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:13.822501 20391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:13.823835 20391 hybrid_clock.cc:648] HybridClock initialized: now 1790549713823763 us; error 59 us; skew 500 ppm
I20260927 22:55:13.824714 20391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:13 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:13.829146 20391 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:55:13.832827 20391 webserver.cc:460] Webserver started at http://127.8.204.129:42325/ using document root <none> and password file <none>
I20260927 22:55:13.833784 20391 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:13.833925 20391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:13.834311 20391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:55:13 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:55:13.851096 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "6e5cb0b945c343da8b04869a0952cf35"
format_stamp: "Formatted at 2026-09-27 22:55:13 on dist-test-slave-40vx"
server_key: "24d7c2b2a4552033d42e46a342c22160"
server_key_iv: "e84f75f416b89569472857341493638e"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:13.852033 20391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "6e5cb0b945c343da8b04869a0952cf35"
format_stamp: "Formatted at 2026-09-27 22:55:13 on dist-test-slave-40vx"
server_key: "24d7c2b2a4552033d42e46a342c22160"
server_key_iv: "e84f75f416b89569472857341493638e"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:13.861059 20391 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260927 22:55:13.890746 20411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:13.893559 20391 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.008s sys 0.007s
I20260927 22:55:13.893805 20391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "6e5cb0b945c343da8b04869a0952cf35"
format_stamp: "Formatted at 2026-09-27 22:55:13 on dist-test-slave-40vx"
server_key: "24d7c2b2a4552033d42e46a342c22160"
server_key_iv: "e84f75f416b89569472857341493638e"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:13.894104 20391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:13.922426 20391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:13.926525 20391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:13.926847 20391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:13.928287 20391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:13.929752 20391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:13.929824 20391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:13.929967 20391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:13.930024 20391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.006372 20391 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41589
I20260927 22:55:14.006382 20523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41589 every 8 connection(s)
I20260927 22:55:14.008160 20391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:55:14.008608 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 20391
I20260927 22:55:14.008817 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:55:14.009835 9010 external_mini_cluster.cc:1428] Setting key 0efde8988e7f0a19fe046c8968e80b4a
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Sep 27 22:55:14 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:55:14.072134 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.018422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:49047 (local address 127.8.204.190:38851)
0927 22:55:14.031960 (+ 13538us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:14.031971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:14.035049 (+ 3078us) server_negotiation.cc:407] Connection header received
0927 22:55:14.035103 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:14.035107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:14.035190 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:14.035460 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:14.039580 (+ 4120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.043831 (+ 4251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.049786 (+ 5955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.050386 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.050526 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:14.061006 (+ 10480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:14.061016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:14.061024 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:14.061084 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:14.065051 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:14.066372 (+ 1321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:14.066381 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:14.066387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:14.066555 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:14.068711 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:14.068719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:14.068725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:14.069156 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:14.069283 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:14.071357 (+ 2074us) server_negotiation.cc:299] Negotiation successful
0927 22:55:14.071625 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13352,"thread_start_us":13253,"threads_started":1}
I20260927 22:55:14.074898 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.017817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:38851 (local address 127.8.204.129:49047)
0927 22:55:14.032899 (+ 15082us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:14.032956 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:14.034761 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:14.035678 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:14.035694 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:14.036971 (+ 1277us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:14.039304 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:14.039341 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.044036 (+ 4695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.044042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:14.049373 (+ 5331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:14.049391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:14.050633 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:14.050639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:14.050892 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:14.053887 (+ 2995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:14.053935 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:14.059631 (+ 5696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:14.065256 (+ 5625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:14.065268 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:14.065289 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:14.065959 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:14.066713 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:14.066720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:14.066727 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:14.067003 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:14.070345 (+ 3342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:14.070356 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:14.070869 (+ 513us) client_negotiation.cc:769] Sending connection context
0927 22:55:14.072686 (+ 1817us) client_negotiation.cc:240] Negotiation successful
0927 22:55:14.073336 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14790,"thread_start_us":14580,"threads_started":1}
I20260927 22:55:14.077224 20524 heartbeater.cc:347] Connected to a master server at 127.8.204.190:38851
I20260927 22:55:14.077548 20524 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:14.078408 20524 heartbeater.cc:511] Master 127.8.204.190:38851 requested a full tablet report, sending...
I20260927 22:55:14.081199 20311 ts_manager.cc:194] Registered new tserver with Master: 6e5cb0b945c343da8b04869a0952cf35 (127.8.204.129:41589)
I20260927 22:55:14.083362 20311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:49047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:55:14.095031 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:42303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38851
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:55:14 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
Sep 27 22:55:14 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
W20260927 22:55:14.610921 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:14.611423 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:14.611614 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:14.622694 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:14.622887 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:14.623102 20530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:55:14.638201 20530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.204.148:42303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:55:14.640128 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:14.641780 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:14.677766 20537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:14.679325 20538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:14.687052 20530 server_base.cc:1034] running on GCE node
W20260927 22:55:14.687469 20541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:14.688292 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:14.694564 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:14.695933 20530 hybrid_clock.cc:648] HybridClock initialized: now 1790549714695835 us; error 105 us; skew 500 ppm
I20260927 22:55:14.696321 20530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:14 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:14.704748 20530 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:55:14.725543 20530 webserver.cc:460] Webserver started at http://127.8.204.130:44981/ using document root <none> and password file <none>
I20260927 22:55:14.726751 20530 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:14.726974 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:14.727443 20530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:55:14 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:55:14.754392 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "b2ad6772ea604ce994da85d7b13ba1a4"
format_stamp: "Formatted at 2026-09-27 22:55:14 on dist-test-slave-40vx"
server_key: "87b359828b98b40a635800e8a02dab56"
server_key_iv: "762da78cb7ca95e4c98ae7e32e51a75c"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:14.756711 20530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "b2ad6772ea604ce994da85d7b13ba1a4"
format_stamp: "Formatted at 2026-09-27 22:55:14 on dist-test-slave-40vx"
server_key: "87b359828b98b40a635800e8a02dab56"
server_key_iv: "762da78cb7ca95e4c98ae7e32e51a75c"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:14.770278 20530 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260927 22:55:14.812213 20554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.814196 20530 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20260927 22:55:14.814474 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "b2ad6772ea604ce994da85d7b13ba1a4"
format_stamp: "Formatted at 2026-09-27 22:55:14 on dist-test-slave-40vx"
server_key: "87b359828b98b40a635800e8a02dab56"
server_key_iv: "762da78cb7ca95e4c98ae7e32e51a75c"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:14.814808 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:14.858999 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:14.863970 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:14.864319 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:14.872004 20530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:14.884171 20530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:14.884336 20530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.884497 20530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:14.884603 20530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:14.968990 20530 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:46597
I20260927 22:55:14.969012 20666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:46597 every 8 connection(s)
I20260927 22:55:14.970546 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:55:14.975639 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 20530
I20260927 22:55:14.975872 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:55:14.976540 9010 external_mini_cluster.cc:1428] Setting key ad9973a8a1b29e2049722ac28a07817c
Sep 27 22:55:15 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
I20260927 22:55:15.026521 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.974390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:50325 (local address 127.8.204.190:38851)
0927 22:55:14.991715 (+ 17325us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:14.991722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.004119 (+ 12397us) server_negotiation.cc:407] Connection header received
0927 22:55:15.007635 (+ 3516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.007641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.007759 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.007866 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.012965 (+ 5099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.013724 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.015331 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.015749 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.015899 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.019472 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.019477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.019480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.019512 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.022507 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.023304 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.023310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.023314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.023467 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.024421 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.024425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.024427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.024757 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.024876 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.025765 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.026067 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17183,"thread_start_us":17083,"threads_started":1}
I20260927 22:55:15.027526 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:14.973850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:38851 (local address 127.8.204.130:50325)
0927 22:55:15.003822 (+ 29972us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.003881 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.007126 (+ 3245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.011624 (+ 4498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.011634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.012130 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:15.012746 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.012765 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.013893 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.013906 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.015155 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.015167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.015983 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.015991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.016228 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.017003 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:15.017027 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:15.019156 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:15.022654 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:15.022667 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:15.022692 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:15.023134 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:15.023630 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:15.023637 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:15.023644 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:15.024293 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:15.024902 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:15.024910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:15.025331 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.025633 (+ 302us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.026154 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29570,"thread_start_us":29337,"threads_started":1}
I20260927 22:55:15.029085 20667 heartbeater.cc:347] Connected to a master server at 127.8.204.190:38851
I20260927 22:55:15.029458 20667 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:15.030447 20667 heartbeater.cc:511] Master 127.8.204.190:38851 requested a full tablet report, sending...
I20260927 22:55:15.032361 20311 ts_manager.cc:194] Registered new tserver with Master: b2ad6772ea604ce994da85d7b13ba1a4 (127.8.204.130:46597)
I20260927 22:55:15.033411 20311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:50325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:55:15.061174 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:42303/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38851
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:55:15.087191 20524 heartbeater.cc:503] Master 127.8.204.190:38851 was elected leader, sending a full tablet report...
W20260927 22:55:15.400574 20673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:55:15.400839 20673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:55:15.400930 20673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:55:15.408507 20673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:55:15.408659 20673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:55:15.408834 20673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:55:15.418793 20673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.204.148:42303/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:38851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:55:15.420159 20673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:55:15.421468 20673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:55:15.434221 20678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:15.434266 20681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:55:15.434180 20679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:55:15.434984 20673 server_base.cc:1034] running on GCE node
I20260927 22:55:15.435672 20673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:55:15.436558 20673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:55:15.437791 20673 hybrid_clock.cc:648] HybridClock initialized: now 1790549715437757 us; error 50 us; skew 500 ppm
I20260927 22:55:15.438506 20673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:55:15 dist-test-slave-40vx krb5kdc[19835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:55:15.441567 20673 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:55:15.442899 20673 webserver.cc:460] Webserver started at http://127.8.204.131:33941/ using document root <none> and password file <none>
I20260927 22:55:15.443614 20673 fs_manager.cc:359] Metadata directory not provided
I20260927 22:55:15.443748 20673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:55:15.444051 20673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:55:15 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:55:15.457854 20673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "7491f6abca00436cbe9c981c22ca9550"
format_stamp: "Formatted at 2026-09-27 22:55:15 on dist-test-slave-40vx"
server_key: "46866be3398dae25493f73541ca578bc"
server_key_iv: "b70fe1021eb381f6b4208c05120273a4"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:15.458709 20673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "7491f6abca00436cbe9c981c22ca9550"
format_stamp: "Formatted at 2026-09-27 22:55:15 on dist-test-slave-40vx"
server_key: "46866be3398dae25493f73541ca578bc"
server_key_iv: "b70fe1021eb381f6b4208c05120273a4"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:15.463901 20673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:55:15.481071 20688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:15.482836 20673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:55:15.483028 20673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "7491f6abca00436cbe9c981c22ca9550"
format_stamp: "Formatted at 2026-09-27 22:55:15 on dist-test-slave-40vx"
server_key: "46866be3398dae25493f73541ca578bc"
server_key_iv: "b70fe1021eb381f6b4208c05120273a4"
server_key_version: "kuduclusterkey@0"
I20260927 22:55:15.483309 20673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:55:15.496645 20673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:55:15.500348 20673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:55:15.500617 20673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:55:15.501463 20673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:55:15.503242 20673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:55:15.503325 20673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 22:55:15.503412 20673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:55:15.503455 20673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:55:15.539917 20673 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36483
I20260927 22:55:15.539958 20800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36483 every 8 connection(s)
I20260927 22:55:15.541134 20673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:55:15.546453 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 20673
I20260927 22:55:15.546655 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:55:15.547245 9010 external_mini_cluster.cc:1428] Setting key 6cac41c913a7840f6315597e368f5296
Sep 27 22:55:15 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549715, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:55:15.561444 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.543681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:46191 (local address 127.8.204.190:38851)
0927 22:55:15.544133 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.544137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.545072 (+ 935us) server_negotiation.cc:407] Connection header received
0927 22:55:15.547409 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.547413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.547494 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.547647 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.549347 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.550263 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.551857 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.552690 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.552841 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.556044 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.556049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.556052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.556091 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.558413 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.559114 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.559119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.559123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.559255 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.559719 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.559724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.559727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.559998 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.560169 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.560820 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.561039 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":230,"threads_started":1}
I20260927 22:55:15.562482 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.544236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:38851 (local address 127.8.204.131:46191)
0927 22:55:15.544892 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.544929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.547065 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.547909 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.547917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.548415 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:55:15.549199 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.549219 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.550409 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.550420 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.551707 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.551720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.552953 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.552962 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.553198 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.553836 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:55:15.553858 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:55:15.555830 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:55:15.558548 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:15.558556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:15.558572 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:15.558985 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:15.559399 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:55:15.559404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:55:15.559408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:55:15.559599 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:55:15.560146 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:55:15.560154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:55:15.560480 (+ 326us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.560735 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.561247 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":278,"threads_started":1}
I20260927 22:55:15.563988 20801 heartbeater.cc:347] Connected to a master server at 127.8.204.190:38851
I20260927 22:55:15.564311 20801 heartbeater.cc:464] Registering TS with master...
I20260927 22:55:15.565027 20801 heartbeater.cc:511] Master 127.8.204.190:38851 requested a full tablet report, sending...
I20260927 22:55:15.566385 20311 ts_manager.cc:194] Registered new tserver with Master: 7491f6abca00436cbe9c981c22ca9550 (127.8.204.131:36483)
I20260927 22:55:15.567325 20311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:46191
I20260927 22:55:15.575203 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:55:15.593279 20803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.579554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.8.204.190:38851)
0927 22:55:15.579741 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.579745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.580264 (+ 519us) server_negotiation.cc:407] Connection header received
0927 22:55:15.580411 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.580415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.580489 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.580574 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.581909 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.582658 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.584155 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.585099 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.585198 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.588159 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.588164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.588167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.588200 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.590916 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.591627 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.591631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.591633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.591750 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.592072 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.592074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.592076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.592345 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.592518 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.592892 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.593060 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:55:15.597581 20311 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:55:15.604885 20311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:55:15.633744 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.618286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:46597 (local address 127.0.0.1:46958)
0927 22:55:15.619164 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.619202 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.619391 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.620646 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.620657 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.620695 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.621249 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.621268 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.624974 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.624982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.630002 (+ 5020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.630026 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.632745 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.632755 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.633038 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.633094 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.633217 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.633367 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":5993,"threads_started":1}
I20260927 22:55:15.634825 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.618477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.8.204.130:46597)
0927 22:55:15.619935 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.619946 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.619975 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:55:15.620071 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.620079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.620393 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.621519 (+ 1126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.621790 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.623983 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.630309 (+ 6326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.632441 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.633985 (+ 1544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.634233 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.634305 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.634477 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1294,"thread_start_us":1162,"threads_started":1}
I20260927 22:55:15.638960 20601 tablet_service.cc:1467] Processing CreateTablet for tablet 1804e2f5d6a444e3a93f3907097769e6 (DEFAULT_TABLE table=test-table [id=e66738d54b684800af967fe1c6c6304f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:15.640260 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1804e2f5d6a444e3a93f3907097769e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:15.641813 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.620483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41589 (local address 127.0.0.1:34286)
0927 22:55:15.633301 (+ 12818us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.633327 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.633491 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.634453 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.634457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.634483 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.634808 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.634823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.636597 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.636602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.638685 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.638699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.640981 (+ 2282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.640987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.641158 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.641191 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.641288 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.641427 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12656,"mutex_wait_us":23,"thread_start_us":12468,"threads_started":1}
I20260927 22:55:15.641866 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.620687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.8.204.129:41589)
0927 22:55:15.632555 (+ 11868us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.632568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.633476 (+ 908us) server_negotiation.cc:407] Connection header received
0927 22:55:15.633677 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.633686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.633980 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.634202 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.635037 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.636446 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.638854 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.640763 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.640895 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.641121 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.641355 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.641513 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11625,"thread_start_us":11875,"threads_started":1}
I20260927 22:55:15.643002 20814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.618419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36483 (local address 127.0.0.1:49864)
0927 22:55:15.624635 (+ 6216us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.624661 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.624797 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.635564 (+ 10767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.635568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.635595 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.635955 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.635966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.637739 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.637747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.639531 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.639546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.642244 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.642250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.642411 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.642441 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.642520 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.642653 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6046,"mutex_wait_us":42,"thread_start_us":357,"threads_started":1}
I20260927 22:55:15.643533 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.619192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.8.204.131:36483)
0927 22:55:15.634769 (+ 15577us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.634776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.634800 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:55:15.634875 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.634881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.635342 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.636176 (+ 834us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.636445 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.637600 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.639713 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.641464 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.641594 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.642872 (+ 1278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.642941 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.643140 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15405,"thread_start_us":15247,"threads_started":1}
I20260927 22:55:15.645664 20459 tablet_service.cc:1467] Processing CreateTablet for tablet 1804e2f5d6a444e3a93f3907097769e6 (DEFAULT_TABLE table=test-table [id=e66738d54b684800af967fe1c6c6304f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:15.647004 20459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1804e2f5d6a444e3a93f3907097769e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:15.647171 20736 tablet_service.cc:1467] Processing CreateTablet for tablet 1804e2f5d6a444e3a93f3907097769e6 (DEFAULT_TABLE table=test-table [id=e66738d54b684800af967fe1c6c6304f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:55:15.648554 20736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1804e2f5d6a444e3a93f3907097769e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:55:15.654577 20820 tablet_bootstrap.cc:492] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Bootstrap starting.
I20260927 22:55:15.658643 20820 tablet_bootstrap.cc:654] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:15.660090 20820 log.cc:824] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:15.660127 20821 tablet_bootstrap.cc:492] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Bootstrap starting.
I20260927 22:55:15.662581 20823 tablet_bootstrap.cc:492] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Bootstrap starting.
I20260927 22:55:15.663635 20821 tablet_bootstrap.cc:654] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:15.663852 20820 tablet_bootstrap.cc:492] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: No bootstrap required, opened a new log
I20260927 22:55:15.664134 20820 ts_tablet_manager.cc:1397] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:55:15.664844 20821 log.cc:824] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:15.666316 20823 tablet_bootstrap.cc:654] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Neither blocks nor log segments found. Creating new log.
I20260927 22:55:15.667392 20821 tablet_bootstrap.cc:492] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: No bootstrap required, opened a new log
I20260927 22:55:15.667488 20823 log.cc:824] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Log is configured to *not* fsync() on all Append() calls
I20260927 22:55:15.667663 20821 ts_tablet_manager.cc:1397] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260927 22:55:15.670817 20823 tablet_bootstrap.cc:492] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: No bootstrap required, opened a new log
I20260927 22:55:15.671053 20823 ts_tablet_manager.cc:1397] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260927 22:55:15.673115 20820 raft_consensus.cc:348] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.673456 20820 raft_consensus.cc:374] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:15.673552 20820 raft_consensus.cc:729] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ad6772ea604ce994da85d7b13ba1a4, State: Initialized, Role: FOLLOWER
I20260927 22:55:15.674140 20820 consensus_queue.cc:260] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.675169 20667 heartbeater.cc:503] Master 127.8.204.190:38851 was elected leader, sending a full tablet report...
I20260927 22:55:15.675676 20820 ts_tablet_manager.cc:1428] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 22:55:15.678314 20821 raft_consensus.cc:348] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.679183 20821 raft_consensus.cc:374] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:15.679306 20821 raft_consensus.cc:729] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5cb0b945c343da8b04869a0952cf35, State: Initialized, Role: FOLLOWER
I20260927 22:55:15.679919 20821 consensus_queue.cc:260] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.680678 20823 raft_consensus.cc:348] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.681026 20823 raft_consensus.cc:374] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:55:15.681136 20823 raft_consensus.cc:729] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7491f6abca00436cbe9c981c22ca9550, State: Initialized, Role: FOLLOWER
I20260927 22:55:15.683581 20821 ts_tablet_manager.cc:1428] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 22:55:15.683946 20823 consensus_queue.cc:260] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.684306 20828 raft_consensus.cc:482] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:55:15.684594 20828 raft_consensus.cc:504] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.685104 20801 heartbeater.cc:503] Master 127.8.204.190:38851 was elected leader, sending a full tablet report...
I20260927 22:55:15.686968 20823 ts_tablet_manager.cc:1428] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260927 22:55:15.687263 20828 leader_election.cc:288] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2ad6772ea604ce994da85d7b13ba1a4 (127.8.204.130:46597), 6e5cb0b945c343da8b04869a0952cf35 (127.8.204.129:41589)
I20260927 22:55:15.692972 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.686816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:46597 (local address 127.8.204.131:59339)
0927 22:55:15.687470 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.687490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.687671 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.688236 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.688240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.688284 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.688497 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.688508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.689657 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.689663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.691130 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.691141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.692220 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.692229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.692420 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.692448 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.692567 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.692696 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541}
I20260927 22:55:15.692965 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.686870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:59339 (local address 127.8.204.130:46597)
0927 22:55:15.687075 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.687083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.687602 (+ 519us) server_negotiation.cc:407] Connection header received
0927 22:55:15.687812 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.687817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.687891 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.687974 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.688665 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.689523 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.691298 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.692070 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.692236 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.692306 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.692591 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.692720 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:55:15.693851 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1804e2f5d6a444e3a93f3907097769e6" candidate_uuid: "7491f6abca00436cbe9c981c22ca9550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" is_pre_election: true
I20260927 22:55:15.694339 20622 raft_consensus.cc:2454] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7491f6abca00436cbe9c981c22ca9550 in term 0.
I20260927 22:55:15.694990 20691 leader_election.cc:302] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7491f6abca00436cbe9c981c22ca9550, b2ad6772ea604ce994da85d7b13ba1a4; no voters:
I20260927 22:55:15.695390 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.688331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:49097 (local address 127.8.204.129:41589)
0927 22:55:15.688507 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.688515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.688565 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:55:15.688826 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.688830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.688896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.688989 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:55:15.690769 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.691611 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.693576 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.694397 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.694576 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.694661 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.694992 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.695116 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:55:15.695405 20828 raft_consensus.cc:2781] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:55:15.695477 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.687759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41589 (local address 127.8.204.131:49097)
0927 22:55:15.688475 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 22:55:15.688492 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:55:15.688665 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:55:15.689002 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:55:15.689006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:55:15.689029 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:55:15.690557 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.690570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.691737 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.691743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.693423 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:55:15.693443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.694602 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.694608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:55:15.694831 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.694867 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:55:15.694959 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:55:15.695092 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":761,"threads_started":1}
I20260927 22:55:15.695585 20828 raft_consensus.cc:482] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:55:15.695693 20828 raft_consensus.cc:3037] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:15.696575 20479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1804e2f5d6a444e3a93f3907097769e6" candidate_uuid: "7491f6abca00436cbe9c981c22ca9550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5cb0b945c343da8b04869a0952cf35" is_pre_election: true
I20260927 22:55:15.697118 20479 raft_consensus.cc:2454] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7491f6abca00436cbe9c981c22ca9550 in term 0.
I20260927 22:55:15.698786 20828 raft_consensus.cc:504] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.699397 20828 leader_election.cc:288] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [CANDIDATE]: Term 1 election: Requested vote from peers b2ad6772ea604ce994da85d7b13ba1a4 (127.8.204.130:46597), 6e5cb0b945c343da8b04869a0952cf35 (127.8.204.129:41589)
I20260927 22:55:15.699826 20622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1804e2f5d6a444e3a93f3907097769e6" candidate_uuid: "7491f6abca00436cbe9c981c22ca9550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2ad6772ea604ce994da85d7b13ba1a4"
I20260927 22:55:15.699867 20479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1804e2f5d6a444e3a93f3907097769e6" candidate_uuid: "7491f6abca00436cbe9c981c22ca9550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e5cb0b945c343da8b04869a0952cf35"
I20260927 22:55:15.700037 20622 raft_consensus.cc:3037] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:15.700054 20479 raft_consensus.cc:3037] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:55:15.702139 20479 raft_consensus.cc:2454] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7491f6abca00436cbe9c981c22ca9550 in term 1.
I20260927 22:55:15.702502 20692 leader_election.cc:302] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e5cb0b945c343da8b04869a0952cf35, 7491f6abca00436cbe9c981c22ca9550; no voters:
I20260927 22:55:15.702744 20622 raft_consensus.cc:2454] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7491f6abca00436cbe9c981c22ca9550 in term 1.
I20260927 22:55:15.702817 20828 raft_consensus.cc:2781] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:55:15.702965 20828 raft_consensus.cc:686] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 1 LEADER]: Becoming Leader. State: Replica: 7491f6abca00436cbe9c981c22ca9550, State: Running, Role: LEADER
I20260927 22:55:15.703542 20828 consensus_queue.cc:237] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } }
I20260927 22:55:15.708483 20310 catalog_manager.cc:5495] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7491f6abca00436cbe9c981c22ca9550 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "7491f6abca00436cbe9c981c22ca9550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7491f6abca00436cbe9c981c22ca9550" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 } health_report { overall_health: UNKNOWN } } }
I20260927 22:55:15.730351 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.724930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.8.204.131:36483)
0927 22:55:15.725141 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.725146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.725190 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:55:15.725376 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.725379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.725457 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.725586 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:55:15.725919 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.726688 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.728252 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.728635 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.728733 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.729212 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:55:15.729752 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:55:15.729858 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.729998 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.730057 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:55:15.739092 20479 raft_consensus.cc:1261] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 1 FOLLOWER]: Refusing update from remote peer 7491f6abca00436cbe9c981c22ca9550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:15.739094 20622 raft_consensus.cc:1261] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 1 FOLLOWER]: Refusing update from remote peer 7491f6abca00436cbe9c981c22ca9550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:55:15.740098 20829 consensus_queue.cc:1035] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2ad6772ea604ce994da85d7b13ba1a4" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 46597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:15.740554 20828 consensus_queue.cc:1035] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e5cb0b945c343da8b04869a0952cf35" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:55:15.751858 20833 mvcc.cc:205] Tried to move back new op lower bound from 7334091635657678848 to 7334091635526377472. Current Snapshot: MvccSnapshot[applied={T|T < 7334091635657678848}]
I20260927 22:55:15.765019 20311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:15.765288 20311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:55:15.768800 20311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc11083b97d640918d6b4fd57105a7e7: Sending DeleteTablet for 3 replicas of tablet 1804e2f5d6a444e3a93f3907097769e6
I20260927 22:55:15.769960 20601 tablet_service.cc:1514] Processing DeleteTablet for tablet 1804e2f5d6a444e3a93f3907097769e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:15 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:46958
I20260927 22:55:15.770020 20736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1804e2f5d6a444e3a93f3907097769e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:15 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49864
I20260927 22:55:15.770159 20459 tablet_service.cc:1514] Processing DeleteTablet for tablet 1804e2f5d6a444e3a93f3907097769e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:55:15 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34286
I20260927 22:55:15.771979 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 20391
I20260927 22:55:15.782436 20844 tablet_replica.cc:330] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: stopping tablet replica
I20260927 22:55:15.783007 20845 tablet_replica.cc:330] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: stopping tablet replica
I20260927 22:55:15.783481 20845 raft_consensus.cc:2229] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:15.783691 20844 raft_consensus.cc:2229] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:55:15.784013 20845 raft_consensus.cc:2258] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:15.784448 20844 raft_consensus.cc:2258] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:15.787050 20844 ts_tablet_manager.cc:1905] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:15.787734 20843 tablet_replica.cc:330] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: stopping tablet replica
I20260927 22:55:15.788166 20843 raft_consensus.cc:2229] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:55:15.788627 20843 raft_consensus.cc:2258] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:55:15.790161 20843 ts_tablet_manager.cc:1905] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:15.792476 20845 ts_tablet_manager.cc:1905] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:55:15.796401 20843 ts_tablet_manager.cc:1918] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:15.796557 20843 log.cc:1196] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/1804e2f5d6a444e3a93f3907097769e6
I20260927 22:55:15.797041 20843 ts_tablet_manager.cc:1939] T 1804e2f5d6a444e3a93f3907097769e6 P b2ad6772ea604ce994da85d7b13ba1a4: Deleting consensus metadata
I20260927 22:55:15.798838 20297 catalog_manager.cc:4841] TS b2ad6772ea604ce994da85d7b13ba1a4 (127.8.204.130:46597): tablet 1804e2f5d6a444e3a93f3907097769e6 (table test-table [id=e66738d54b684800af967fe1c6c6304f]) successfully deleted
I20260927 22:55:15.800822 20845 ts_tablet_manager.cc:1918] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:15.801041 20845 log.cc:1196] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/1804e2f5d6a444e3a93f3907097769e6
I20260927 22:55:15.801549 20845 ts_tablet_manager.cc:1939] T 1804e2f5d6a444e3a93f3907097769e6 P 6e5cb0b945c343da8b04869a0952cf35: Deleting consensus metadata
I20260927 22:55:15.802548 20844 ts_tablet_manager.cc:1918] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:55:15.802712 20844 log.cc:1196] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/1804e2f5d6a444e3a93f3907097769e6
I20260927 22:55:15.803299 20844 ts_tablet_manager.cc:1939] T 1804e2f5d6a444e3a93f3907097769e6 P 7491f6abca00436cbe9c981c22ca9550: Deleting consensus metadata
I20260927 22:55:15.803440 20298 catalog_manager.cc:4841] TS 6e5cb0b945c343da8b04869a0952cf35 (127.8.204.129:41589): tablet 1804e2f5d6a444e3a93f3907097769e6 (table test-table [id=e66738d54b684800af967fe1c6c6304f]) successfully deleted
I20260927 22:55:15.804890 20295 catalog_manager.cc:4841] TS 7491f6abca00436cbe9c981c22ca9550 (127.8.204.131:36483): tablet 1804e2f5d6a444e3a93f3907097769e6 (table test-table [id=e66738d54b684800af967fe1c6c6304f]) successfully deleted
Sep 27 22:55:15 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:55:15.818365 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.773425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.8.204.129:41589)
0927 22:55:15.773611 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.773618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.796470 (+ 22852us) server_negotiation.cc:407] Connection header received
0927 22:55:15.796827 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.796833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.796916 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.797025 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.799461 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.800583 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.802500 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.803213 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.803340 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.809948 (+ 6608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.809968 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.810037 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.810118 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.814984 (+ 4866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.816016 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.816032 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.816034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.816185 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.816652 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.816656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.816659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.817078 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.817324 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.817751 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.817945 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:55:15.819171 20519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:15.911461 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 20391
I20260927 22:55:15.934917 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 20530
Sep 27 22:55:15 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:55:15.952579 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:15.935942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.8.204.130:46597)
0927 22:55:15.936117 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:15.936124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:15.936148 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:55:15.936304 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:15.936310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:15.936402 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:15.936517 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:15.937889 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.938952 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.940570 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:15.941207 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:15.941346 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:15.945593 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:15.945610 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:15.945631 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:15.945672 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:15.949751 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.950433 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.950442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.950446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.950605 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:15.950971 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:15.950976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:15.950980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:15.951462 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:15.951753 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:15.951978 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:55:15.952198 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:55:15.953362 20662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:16.047379 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 20530
I20260927 22:55:16.063856 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 20673
Sep 27 22:55:16 dist-test-slave-40vx krb5kdc[19835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:55:16.080473 20817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:55:16.064816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.8.204.131:36483)
0927 22:55:16.065014 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:55:16.065023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:55:16.065043 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:55:16.065113 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:55:16.065131 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:55:16.065203 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:55:16.065327 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:55:16.066630 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:16.067487 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:16.068774 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:55:16.069226 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:55:16.069363 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:55:16.074311 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:55:16.074322 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:55:16.074337 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:55:16.074371 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:55:16.077417 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:16.078282 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:16.078289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:16.078291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:16.078443 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:55:16.078896 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:55:16.078899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:55:16.078901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:55:16.079212 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:55:16.079585 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 22:55:16.080002 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 22:55:16.080168 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:55:16.081142 20796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:16.204481 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 20673
I20260927 22:55:16.224352 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 20279
I20260927 22:55:16.225574 20341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:55:16.338768 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 20279
I20260927 22:55:16.362519 9010 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:55:16.697678 9010 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:55:16.697878 9010 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:55:17.043642 9010 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:55:17.044 UTC [19865] LOG: received smart shutdown request
2026-09-27 22:55:17.047 UTC [19865] LOG: background worker "logical replication launcher" (PID 19874) exited with exit code 1
2026-09-27 22:55:17.048 UTC [19869] LOG: shutting down
2026-09-27 22:55:17.222 UTC [19865] LOG: database system is shut down
2026-09-27T22:55:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48163 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:55:17 dist-test-slave-40vx krb5kdc[20859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:55:17 dist-test-slave-40vx krb5kdc[20859](info): set up 2 sockets
Sep 27 22:55:17 dist-test-slave-40vx krb5kdc[20859](info): commencing operation
krb5kdc: starting...
W20260927 22:55:18.447662 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:55:18 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:55:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:55:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/rangeradmin_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/rangeradmin_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/rangerlookup_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/rangerlookup_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
I20260927 22:55:18.756453 9010 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskoq4u9H/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/postgres -l logfile start
2026-09-27 22:55:21.290 UTC [20889] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:55:21.291 UTC [20889] LOG: listening on IPv4 address "127.8.204.148", port 38219
2026-09-27 22:55:21.294 UTC [20889] LOG: listening on Unix socket "/tmp/.s.PGSQL.38219"
2026-09-27 22:55:21.318 UTC [20892] LOG: database system was shut down at 2026-09-27 22:55:19 UTC
2026-09-27 22:55:21.322 UTC [20889] LOG: database system is ready to accept connections
I20260927 22:55:22.315363 9010 mini_postgres.cc:96] Postgres bound to 38219
2026-09-27 22:55:22.338 UTC [20900] FATAL: database "slave" does not exist
127.8.204.148:38219 - accepting connections
I20260927 22:55:22.339150 9010 mini_ranger.cc:161] Starting Ranger...
I20260927 22:55:22.364494 9010 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:55:22.873282 9010 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:55:22.873406 9010 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-admin
2026-09-27 22:55:23,156 [I] DB FLAVOR :POSTGRES
2026-09-27 22:55:23,157 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:55:23,157 [I] Checking connection..
2026-09-27 22:55:23,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:55:23,417 [I] Checking connection passed.
2026-09-27 22:55:23,418 [I] --------- Verifying version history table ---------
2026-09-27 22:55:23,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:55:23,681 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:55:23,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:55:23,927 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:55:23,928 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:55:23,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:55:24.164 UTC [20964] WARNING: there is no transaction in progress
2026-09-27 22:55:24,170 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:55:24,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:55:24,422 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:55:24,422 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:55:24,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:55:24,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:55:24,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-40vx.c.gcp-upstream.internal','N') ;"
2026-09-27 22:55:25,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:55:25,393 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:55:25,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:55:25,630 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:55:25,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:55:25,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:55:26,111 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:55:26,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:55:27.063 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.080 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.095 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.109 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.126 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.144 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.203 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.210 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.225 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.235 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.245 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.258 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.267 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.273 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.280 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27.286 UTC [21072] WARNING: there is no transaction in progress
2026-09-27 22:55:27,946 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:55:27,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:55:28,197 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:55:28,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:55:28,435 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:55:28,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:55:28,693 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:55:28,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:55:28,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:55:29,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-40vx.c.gcp-upstream.internal';"
2026-09-27 22:55:29,394 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:55:29,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:55:29,639 [I] DB_PATCHES have already been applied
I20260927 22:55:29.645134 9010 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-admin:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskoq4u9H/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:55:29.645224 9010 mini_ranger.cc:193] Using host: 127.8.204.148
I20260927 22:55:29.659157 9010 mini_ranger.cc:227] Ranger admin URL: http://127.8.204.148:52575
Sep 27, 2026 10:55:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:55:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:55:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:55:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:55:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:55:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:55:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:55:30.789059 9010 mini_ranger.cc:160] Time spent starting Ranger: real 8.450s user 0.002s sys 0.035s
Sep 27, 2026 10:55:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:55:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:55:51 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:55:51 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:55:52 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:55:52.600827 9010 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.204.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.204.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/rangerkms_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.204.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/rangerkms_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.204.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.204.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.204.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/HTTP_127.8.204.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:55:52 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549752, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:55:52.759905 9010 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:55:52.780424 9010 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:55:53.679237 9010 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:55:53.679409 9010 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms
I20260927 22:55:53.679466 9010 mini_ranger_kms.cc:227] Using postgres at 127.8.204.148:38219
2026-09-27 22:55:54,035 [I] DB FLAVOR :POSTGRES
2026-09-27 22:55:54,036 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:55:54,036 [I] Checking connection
2026-09-27 22:55:54,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:55:54,284 [I] connection success
2026-09-27 22:55:54,284 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:55:54,285 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:55:54,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:55:54,520 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:55:54,520 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:55:54,521 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:55:54,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.204.148:38219/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:55:54,791 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:55:55.245743 9010 mini_ranger_kms.cc:310] Created kms service
I20260927 22:55:55.430011 9010 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:55:55.474579 9010 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:55:55 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549752, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:55:55.882019 9010 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:55:55.882357 9010 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms:/tmp/dist-test-taskoq4u9H/build/asan/bin/postgresql.jar:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskoq4u9H/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:55:55.882438 9010 mini_ranger_kms.cc:240] Using host: 127.8.204.148
I20260927 22:55:55.899549 9010 mini_ranger_kms.cc:276] Ranger KMS PID: 21252
I20260927 22:55:55.899688 9010 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.204.148:60197
Sep 27, 2026 10:55:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:55:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:55:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:55:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:55:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:55:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:55:58.177837 9010 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.418s user 0.085s sys 0.290s
I20260927 22:55:58.178084 9010 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:55:58.178264 9010 mini_ranger_kms.cc:401] 127.8.204.148:60197/kms/v1/keys
Sep 27 22:55:58 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549758, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:55:58 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549758, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.148@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
Sep 27 22:55:59 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549758, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.204.148@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
Sep 27 22:56:01 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:56:01.288968 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42333
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:42333
--ranger_config_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:01.622335 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:01.622605 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:01.622689 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:01.629925 21303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:01.630030 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:01.630076 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:01.630131 21303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:01.630178 21303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:01.640069 21303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.204.148:60197/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:42333
--ranger_config_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:01.641294 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:01.642555 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:01.654289 21311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:01.654384 21309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:01.654328 21308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:01.654764 21303 server_base.cc:1034] running on GCE node
I20260927 22:56:01.655665 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:01.657060 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:01.658305 21303 hybrid_clock.cc:648] HybridClock initialized: now 1790549761658263 us; error 32 us; skew 500 ppm
I20260927 22:56:01.658669 21303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:01 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:01.661715 21303 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:01.663089 21303 webserver.cc:460] Webserver started at http://127.8.204.190:38385/ using document root <none> and password file <none>
I20260927 22:56:01.663784 21303 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:01.663924 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:01.664223 21303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:56:01 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:01.728792 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "49cce94fa47d48df83027658a1a31d47"
format_stamp: "Formatted at 2026-09-27 22:56:01 on dist-test-slave-40vx"
server_key: "2bc7c8f6a7e75ec7c68f5d02b9788f88"
server_key_iv: "c84612b5e021f8ae80a62d32f6aaa39a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:01.729660 21303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "49cce94fa47d48df83027658a1a31d47"
format_stamp: "Formatted at 2026-09-27 22:56:01 on dist-test-slave-40vx"
server_key: "2bc7c8f6a7e75ec7c68f5d02b9788f88"
server_key_iv: "c84612b5e021f8ae80a62d32f6aaa39a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:01.735128 21303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:56:01.755213 21318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:01.756920 21303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:56:01.757108 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "49cce94fa47d48df83027658a1a31d47"
format_stamp: "Formatted at 2026-09-27 22:56:01 on dist-test-slave-40vx"
server_key: "2bc7c8f6a7e75ec7c68f5d02b9788f88"
server_key_iv: "c84612b5e021f8ae80a62d32f6aaa39a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:01.757323 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:01.787894 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:01.792272 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:01.792598 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:01.818441 21303 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:42333
I20260927 22:56:01.818470 21369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:42333 every 8 connection(s)
I20260927 22:56:01.820309 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:01.824568 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 21303
I20260927 22:56:01.824785 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:56:01.825328 21370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:01.825625 9010 external_mini_cluster.cc:1428] Setting key 01ede2dc8dcd74edeca577289352a5a2
I20260927 22:56:01.838778 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Bootstrap starting.
I20260927 22:56:01.841711 21370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:01.842664 21370 log.cc:824] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:01.845242 21370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: No bootstrap required, opened a new log
Sep 27 22:56:01 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:01.852888 21370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:01.853358 21370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:01.853484 21370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cce94fa47d48df83027658a1a31d47, State: Initialized, Role: FOLLOWER
I20260927 22:56:01.854063 21370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:01.854349 21370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:01.854501 21370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:01.854691 21370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:01.856933 21370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:01.857446 21370 leader_election.cc:302] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49cce94fa47d48df83027658a1a31d47; no voters:
I20260927 22:56:01.858106 21370 leader_election.cc:288] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:01.858229 21375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:01.859328 21375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 LEADER]: Becoming Leader. State: Replica: 49cce94fa47d48df83027658a1a31d47, State: Running, Role: LEADER
I20260927 22:56:01.859647 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:01.827690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.8.204.190:42333)
0927 22:56:01.828524 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:01.828535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:01.828576 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:56:01.830149 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:01.830170 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:01.830547 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:01.830877 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:01.832501 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:01.833716 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:01.835664 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:01.836341 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:01.836482 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:01.851237 (+ 14755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:01.851246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:01.851263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:01.851369 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:01.854822 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:01.855651 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:01.855660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:01.855669 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:01.855828 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:01.856234 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:01.856239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:01.856243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:01.856775 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:01.857107 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:01.857533 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:56:01.857999 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":403,"threads_started":1}
I20260927 22:56:01.859967 21375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:01.860390 21370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:01.862741 21377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49cce94fa47d48df83027658a1a31d47. Latest consensus state: current_term: 1 leader_uuid: "49cce94fa47d48df83027658a1a31d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } } }
I20260927 22:56:01.863148 21377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:01.864856 21376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49cce94fa47d48df83027658a1a31d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } } }
I20260927 22:56:01.865239 21376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:01.865435 21380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:01.871466 21380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:01.872145 21370 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:56:01.894809 21380 catalog_manager.cc:1347] Generated new cluster ID: d0c586d861564d2da3d61faad90a8d89
I20260927 22:56:01.895035 21380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:01.921175 21380 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:01.923250 21380 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:01.934172 21380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Generated new TSK 0
I20260927 22:56:01.935315 21380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:56:02.642876 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:42333
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:56:02 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:56:03.046669 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:03.047012 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:03.047216 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:03.055649 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:03.055794 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:03.056010 21415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:56:03.068539 21415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.204.148:60197/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:03.069932 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:03.071257 21415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:03.088095 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.089295 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:03.095757 21415 server_base.cc:1034] running on GCE node
W20260927 22:56:03.096696 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:03.097539 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:03.098924 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:03.100225 21415 hybrid_clock.cc:648] HybridClock initialized: now 1790549763100105 us; error 109 us; skew 500 ppm
I20260927 22:56:03.100623 21415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:03 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:03.104892 21415 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:56:03.111689 21415 webserver.cc:460] Webserver started at http://127.8.204.129:41551/ using document root <none> and password file <none>
I20260927 22:56:03.112653 21415 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:03.112793 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:03.113153 21415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:56:03 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:03.135483 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "ede4c749c18846048519f3355db69abb"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-40vx"
server_key: "f2965601849e1ba722b7a372199702ee"
server_key_iv: "07043132403ce69d25476b3d7080bdeb"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:03.136837 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "ede4c749c18846048519f3355db69abb"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-40vx"
server_key: "f2965601849e1ba722b7a372199702ee"
server_key_iv: "07043132403ce69d25476b3d7080bdeb"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:03.144484 21415 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 22:56:03.175933 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.178143 21415 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.007s
I20260927 22:56:03.178336 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "ede4c749c18846048519f3355db69abb"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-40vx"
server_key: "f2965601849e1ba722b7a372199702ee"
server_key_iv: "07043132403ce69d25476b3d7080bdeb"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:03.178603 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:03.239174 21415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:03.244331 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:03.244685 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:03.245754 21415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:03.247344 21415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:03.247466 21415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.247633 21415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:03.247712 21415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:03.330600 21415 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:44897
I20260927 22:56:03.330634 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:44897 every 8 connection(s)
I20260927 22:56:03.332211 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:03.339921 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 21415
I20260927 22:56:03.340145 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:03.341065 9010 external_mini_cluster.cc:1428] Setting key d8bc7c2baeb4318d089d895833bd28c4
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Sep 27 22:56:03 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:56:03.411713 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.339406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:44591 (local address 127.8.204.190:42333)
0927 22:56:03.368567 (+ 29161us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:03.368578 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:03.375295 (+ 6717us) server_negotiation.cc:407] Connection header received
0927 22:56:03.378419 (+ 3124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:03.378427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:03.378548 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:03.378649 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:03.382787 (+ 4138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.384385 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.392730 (+ 8345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.393472 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.393595 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.403219 (+ 9624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:03.403227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:03.403231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:03.403319 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:03.406998 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.408083 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.408089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.408093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.408243 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:03.408833 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:03.408839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:03.408842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:03.409211 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:03.409354 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:03.411004 (+ 1650us) server_negotiation.cc:299] Negotiation successful
0927 22:56:03.411241 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28972,"thread_start_us":29164,"threads_started":1}
I20260927 22:56:03.413014 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:03.338491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42333 (local address 127.8.204.129:44591)
0927 22:56:03.372451 (+ 33960us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:03.372517 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:03.377997 (+ 5480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:03.379380 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:03.379399 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:03.381560 (+ 2161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:03.382544 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.382575 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.384625 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.384631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.392211 (+ 7580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:03.392231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:03.394000 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:03.394008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:03.394327 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:03.395200 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:03.395235 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:03.402869 (+ 7634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:03.407174 (+ 4305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:03.407186 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:03.407207 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:03.407920 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:03.408376 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:03.408382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:03.408386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:03.408689 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:03.409828 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:03.409837 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:03.410353 (+ 516us) client_negotiation.cc:769] Sending connection context
0927 22:56:03.410716 (+ 363us) client_negotiation.cc:240] Negotiation successful
0927 22:56:03.411352 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33517,"thread_start_us":33745,"threads_started":1}
I20260927 22:56:03.413728 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:42333
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:03.420158 21548 heartbeater.cc:347] Connected to a master server at 127.8.204.190:42333
I20260927 22:56:03.420509 21548 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:03.421375 21548 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:03.424073 21335 ts_manager.cc:194] Registered new tserver with Master: ede4c749c18846048519f3355db69abb (127.8.204.129:44897)
I20260927 22:56:03.426200 21335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:44591
Sep 27 22:56:03 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
Sep 27 22:56:03 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549762, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
W20260927 22:56:03.920028 21554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:03.920606 21554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:03.920825 21554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:03.933830 21554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:03.934082 21554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:03.934326 21554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:56:03.951720 21554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.204.148:60197/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:03.953572 21554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:03.955500 21554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:03.979494 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.980276 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:03.988216 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:03.988875 21554 server_base.cc:1034] running on GCE node
I20260927 22:56:03.989650 21554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:03.990895 21554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:03.992194 21554 hybrid_clock.cc:648] HybridClock initialized: now 1790549763992129 us; error 50 us; skew 500 ppm
I20260927 22:56:03.992573 21554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:03 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:03.996978 21554 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:56:04.017431 21554 webserver.cc:460] Webserver started at http://127.8.204.130:41585/ using document root <none> and password file <none>
I20260927 22:56:04.018656 21554 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:04.018908 21554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:04.019429 21554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:04.061424 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "029e074423a942b8ae1c0e5c2a88f7c9"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "31cc57e243e5e9f80e114c6be80f4f16"
server_key_iv: "5e0dd1bc3db4921a1b5386d8496e8d16"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:04.062721 21554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "029e074423a942b8ae1c0e5c2a88f7c9"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "31cc57e243e5e9f80e114c6be80f4f16"
server_key_iv: "5e0dd1bc3db4921a1b5386d8496e8d16"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:04.084611 21554 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.010s sys 0.007s
I20260927 22:56:04.114418 21578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:04.116686 21554 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.001s
I20260927 22:56:04.116885 21554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "029e074423a942b8ae1c0e5c2a88f7c9"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "31cc57e243e5e9f80e114c6be80f4f16"
server_key_iv: "5e0dd1bc3db4921a1b5386d8496e8d16"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:04.117158 21554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:04.151816 21554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:04.157058 21554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:04.157442 21554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:04.158674 21554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:04.160660 21554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:04.160781 21554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:04.160921 21554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:04.161003 21554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:04.201875 21554 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:45311
I20260927 22:56:04.201893 21690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:45311 every 8 connection(s)
I20260927 22:56:04.203503 21554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:04.209154 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 21554
I20260927 22:56:04.209334 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:04.209761 9010 external_mini_cluster.cc:1428] Setting key 1be67dc869cfc3d2243b6641c225653c
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549763, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:04.226563 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.206099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:35229 (local address 127.8.204.190:42333)
0927 22:56:04.206573 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.206580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.207658 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 22:56:04.210110 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.210116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.210202 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.210330 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.212402 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.213654 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.216054 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.216789 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.216931 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.220707 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.220712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.220715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.220746 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.223634 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.224334 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.224339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.224343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.224463 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.224905 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.224910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.224913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.225174 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.225327 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.225963 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.226184 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":223,"threads_started":1}
I20260927 22:56:04.227586 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.206663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42333 (local address 127.8.204.130:35229)
0927 22:56:04.207484 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:04.207533 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:04.209803 (+ 2270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:04.210597 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:04.210608 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:04.211188 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:04.212225 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.212248 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.213870 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.213876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.215899 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.215913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.217018 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.217025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.217294 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.218067 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:04.218095 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:04.220491 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:04.223759 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:04.223785 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:04.223800 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:04.224211 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:04.224577 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:04.224583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:04.224587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:04.224789 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:04.225297 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:04.225305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:04.225613 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 22:56:04.225860 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 22:56:04.226343 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":303,"threads_started":1}
I20260927 22:56:04.228921 21691 heartbeater.cc:347] Connected to a master server at 127.8.204.190:42333
I20260927 22:56:04.229223 21691 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:04.229941 21691 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:04.231482 21335 ts_manager.cc:194] Registered new tserver with Master: 029e074423a942b8ae1c0e5c2a88f7c9 (127.8.204.130:45311)
I20260927 22:56:04.232231 21335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:35229
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:56:04.266258 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:42333
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:04.430807 21548 heartbeater.cc:503] Master 127.8.204.190:42333 was elected leader, sending a full tablet report...
W20260927 22:56:04.593278 21697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:04.593539 21697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:04.593689 21697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:04.600852 21697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:04.600975 21697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:04.601150 21697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:56:04.611420 21697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.204.148:60197/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:04.612672 21697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:04.613927 21697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:04.626665 21702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:04.626709 21703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:04.626637 21705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:04.628059 21697 server_base.cc:1034] running on GCE node
I20260927 22:56:04.628564 21697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:04.629460 21697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:04.630649 21697 hybrid_clock.cc:648] HybridClock initialized: now 1790549764630605 us; error 35 us; skew 500 ppm
I20260927 22:56:04.630956 21697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549764, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:04.633914 21697 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:56:04.635349 21697 webserver.cc:460] Webserver started at http://127.8.204.131:35689/ using document root <none> and password file <none>
I20260927 22:56:04.636050 21697 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:04.636188 21697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:04.636498 21697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549764, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:04.652845 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "75ed027212fa40c488cf8069e5e9d68c"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "08c80dbf65446166d9bd2fc1bcd3faf1"
server_key_iv: "efee79a947e2f2cf2ed41d56e2a58c1a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:04.653887 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "75ed027212fa40c488cf8069e5e9d68c"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "08c80dbf65446166d9bd2fc1bcd3faf1"
server_key_iv: "efee79a947e2f2cf2ed41d56e2a58c1a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:04.660303 21697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 22:56:04.676870 21712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:04.678561 21697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:04.678730 21697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "75ed027212fa40c488cf8069e5e9d68c"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "08c80dbf65446166d9bd2fc1bcd3faf1"
server_key_iv: "efee79a947e2f2cf2ed41d56e2a58c1a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:04.678974 21697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:04.711082 21697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:04.714819 21697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:04.715070 21697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:04.716094 21697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:04.717370 21697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:04.717440 21697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:04.717519 21697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:04.717566 21697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:04.750442 21697 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36093
I20260927 22:56:04.750483 21824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36093 every 8 connection(s)
I20260927 22:56:04.751752 21697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:04.761245 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 21697
I20260927 22:56:04.761461 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:04.762039 9010 external_mini_cluster.cc:1428] Setting key 22e227954f6e4b4cf39705eb96f9d0db
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549764, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:04.770783 21827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.754256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:52191 (local address 127.8.204.190:42333)
0927 22:56:04.754675 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.754679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.755693 (+ 1014us) server_negotiation.cc:407] Connection header received
0927 22:56:04.757447 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.757453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.757545 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.757668 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.759066 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.759846 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.761512 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.762180 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.762343 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.765657 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.765662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.765665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.765701 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.768380 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.769024 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.769028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.769030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.769143 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.769484 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.769487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.769489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.769718 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.769833 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.770388 (+ 555us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.770545 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":195,"threads_started":1}
I20260927 22:56:04.771786 21828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.754799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42333 (local address 127.8.204.131:52191)
0927 22:56:04.755531 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:04.755574 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:04.757174 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:04.757901 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:04.757909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:04.758341 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:04.758902 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.758919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.759993 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.760000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.761327 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:04.761345 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.762396 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.762402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:04.762610 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.763403 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:04.763431 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:04.765479 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:04.768509 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:04.768516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:04.768527 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:04.768923 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:04.769231 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:04.769234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:04.769236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:04.769389 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:04.769823 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:04.769828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:04.770102 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 22:56:04.770318 (+ 216us) client_negotiation.cc:240] Negotiation successful
0927 22:56:04.770670 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":272,"threads_started":1}
I20260927 22:56:04.773037 21825 heartbeater.cc:347] Connected to a master server at 127.8.204.190:42333
I20260927 22:56:04.773280 21825 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:04.773870 21825 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:04.775146 21335 ts_manager.cc:194] Registered new tserver with Master: 75ed027212fa40c488cf8069e5e9d68c (127.8.204.131:36093)
I20260927 22:56:04.776213 21335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:52191
I20260927 22:56:04.777616 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:04.779251 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 21415
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:56:04.794998 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.780207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34318 (local address 127.8.204.129:44897)
0927 22:56:04.780791 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.780796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.780819 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:56:04.780871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.780877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.781051 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.781178 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.782454 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.783540 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.785041 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.785612 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.785705 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.789886 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.789896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.789931 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.789968 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.792598 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.793282 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.793289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.793291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.793446 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.793751 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.793755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.793757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.794061 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.794284 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.794539 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.794722 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":292,"threads_started":1}
I20260927 22:56:04.796482 21543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:04.880339 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 21415
I20260927 22:56:04.896947 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 21554
Sep 27 22:56:04 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:56:04.912487 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:04.897876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.8.204.130:45311)
0927 22:56:04.898407 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:04.898415 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:04.898441 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:56:04.898500 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:04.898506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:04.898720 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:04.898845 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:04.900080 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.901068 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.902389 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:04.902915 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:04.903016 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:04.907039 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:04.907050 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:04.907063 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:04.907096 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:04.909900 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.910504 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.910510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.910512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.910631 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:04.910937 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:04.910941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:04.910943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:04.911245 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:04.911527 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:04.911989 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 22:56:04.912177 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":286,"threads_started":1}
I20260927 22:56:04.913676 21686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:04.995362 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 21554
I20260927 22:56:05.011622 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 21697
Sep 27 22:56:05 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:56:05.028014 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:05.012774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.8.204.131:36093)
0927 22:56:05.013413 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:05.013420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:05.013461 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:56:05.013521 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:05.013528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:05.013733 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:05.013860 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:05.015147 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.016221 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.017559 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:05.018071 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:05.018161 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:05.022529 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:05.022538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:05.022552 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:05.022586 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:05.025405 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:05.026051 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:05.026058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:05.026060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:05.026179 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:05.026508 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:05.026511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:05.026513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:05.026812 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:05.027054 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:05.027399 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 22:56:05.027685 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":344,"threads_started":1}
I20260927 22:56:05.029356 21820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:05.112567 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 21697
I20260927 22:56:05.128116 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 21303
I20260927 22:56:05.129383 21365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:05.236160 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 21303
I20260927 22:56:05.257584 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42333
--webserver_interface=127.8.204.190
--webserver_port=38385
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:42333
--ranger_config_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:05.575860 21837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:05.576196 21837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:05.576362 21837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:05.583448 21837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:05.583603 21837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:05.583702 21837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:05.583798 21837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:05.583889 21837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:05.593953 21837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:42333
--ranger_config_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=38385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:05.595225 21837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:05.596591 21837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:05.608333 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:05.608405 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:05.608361 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:05.609082 21837 server_base.cc:1034] running on GCE node
I20260927 22:56:05.609751 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:05.611078 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:05.612277 21837 hybrid_clock.cc:648] HybridClock initialized: now 1790549765612228 us; error 49 us; skew 500 ppm
I20260927 22:56:05.612583 21837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:05 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:05.615628 21837 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:05.617069 21837 webserver.cc:460] Webserver started at http://127.8.204.190:38385/ using document root <none> and password file <none>
I20260927 22:56:05.617694 21837 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:05.617846 21837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:56:05 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549765, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:05.671677 21837 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:05.674723 21853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:05.676141 21837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:05.676332 21837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "49cce94fa47d48df83027658a1a31d47"
format_stamp: "Formatted at 2026-09-27 22:56:01 on dist-test-slave-40vx"
server_key: "2bc7c8f6a7e75ec7c68f5d02b9788f88"
server_key_iv: "c84612b5e021f8ae80a62d32f6aaa39a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:05.677294 21837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:05.700270 21837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:05.703940 21837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:05.704221 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:05.722443 21837 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:42333
I20260927 22:56:05.722466 21904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:42333 every 8 connection(s)
I20260927 22:56:05.723922 21837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:05.730090 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 21837
I20260927 22:56:05.730855 21905 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:56:05.731148 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:44897
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:42333
--webserver_port=41551
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:05.736066 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Bootstrap starting.
I20260927 22:56:05.755155 21905 log.cc:824] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:05.763978 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:56:05.764484 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Bootstrap complete.
I20260927 22:56:05.773247 21905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:05.774034 21905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49cce94fa47d48df83027658a1a31d47, State: Initialized, Role: FOLLOWER
I20260927 22:56:05.774621 21905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:05.774832 21905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:05.774966 21905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:05.775141 21905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:56:05.778764 21905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:05.779217 21905 leader_election.cc:302] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49cce94fa47d48df83027658a1a31d47; no voters:
I20260927 22:56:05.779850 21905 leader_election.cc:288] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:56:05.779982 21909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:56:05.781500 21909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [term 2 LEADER]: Becoming Leader. State: Replica: 49cce94fa47d48df83027658a1a31d47, State: Running, Role: LEADER
I20260927 22:56:05.782068 21905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:05.782112 21909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } }
I20260927 22:56:05.784610 21911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49cce94fa47d48df83027658a1a31d47. Latest consensus state: current_term: 2 leader_uuid: "49cce94fa47d48df83027658a1a31d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } } }
I20260927 22:56:05.784965 21911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:05.785480 21910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "49cce94fa47d48df83027658a1a31d47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49cce94fa47d48df83027658a1a31d47" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42333 } } }
I20260927 22:56:05.785766 21910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:05.803877 21915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:05.809710 21915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:05.811051 21915 catalog_manager.cc:1259] Loaded cluster ID: d0c586d861564d2da3d61faad90a8d89
I20260927 22:56:05.811357 21915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:05.820227 21915 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:05.822017 21915 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Loaded TSK: 0
I20260927 22:56:05.824309 21915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:05.828248 21905 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 22:56:06.178550 21907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:06.179040 21907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:06.179208 21907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:06.191184 21907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:06.191344 21907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:06.191582 21907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:56:06.207099 21907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:44897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=41551
--tserver_master_addrs=127.8.204.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:06.208877 21907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:06.210538 21907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:06.257820 21942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:06.262274 21943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:06.264439 21907 server_base.cc:1034] running on GCE node
W20260927 22:56:06.264968 21946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:06.265787 21907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:06.267917 21907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:06.271304 21907 hybrid_clock.cc:648] HybridClock initialized: now 1790549766271220 us; error 58 us; skew 500 ppm
I20260927 22:56:06.271715 21907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:06 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549766, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:06.276623 21907 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:56:06.305982 21907 webserver.cc:460] Webserver started at http://127.8.204.129:41551/ using document root <none> and password file <none>
I20260927 22:56:06.307157 21907 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:06.307447 21907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:56:06 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549766, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:06.349996 21907 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.003s sys 0.007s
I20260927 22:56:06.363893 21954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.369463 21907 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.003s
I20260927 22:56:06.369750 21907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "ede4c749c18846048519f3355db69abb"
format_stamp: "Formatted at 2026-09-27 22:56:03 on dist-test-slave-40vx"
server_key: "f2965601849e1ba722b7a372199702ee"
server_key_iv: "07043132403ce69d25476b3d7080bdeb"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:06.371322 21907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:06.425223 21907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:06.430335 21907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:06.430724 21907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:06.439069 21907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:06.440768 21907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:06.440868 21907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.441018 21907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:06.441105 21907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:06.527324 21907 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:44897
I20260927 22:56:06.527392 22066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:44897 every 8 connection(s)
I20260927 22:56:06.528863 21907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:06.537549 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 21907
I20260927 22:56:06.538244 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:45311
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:42333
--webserver_port=41585
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:56:06 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549766, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:06.564563 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.532382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42333 (local address 127.8.204.129:49469)
0927 22:56:06.538045 (+ 5663us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:06.538103 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:06.540620 (+ 2517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:06.543301 (+ 2681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:06.543313 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:06.543970 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:06.544773 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.544800 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.547216 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.547221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.549279 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:06.549293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.550793 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.550800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:06.551064 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.551897 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:06.551928 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:06.554755 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:06.559251 (+ 4496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:06.559260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:06.559308 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:06.559937 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:06.560489 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:06.560495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:06.560500 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:06.560722 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:06.561837 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:06.561845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:06.562242 (+ 397us) client_negotiation.cc:769] Sending connection context
0927 22:56:06.562568 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 22:56:06.563114 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5359,"thread_start_us":13214,"threads_started":1}
I20260927 22:56:06.566912 22069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:06.532344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:49469 (local address 127.8.204.190:42333)
0927 22:56:06.537456 (+ 5112us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:06.537477 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:06.538280 (+ 803us) server_negotiation.cc:407] Connection header received
0927 22:56:06.541557 (+ 3277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:06.541578 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:06.542105 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:06.542555 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:06.545016 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.547011 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.549442 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:06.550540 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:06.550736 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:06.555056 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:06.555066 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:06.555084 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:06.555127 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:06.559071 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:06.560111 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:06.560120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:06.560128 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:06.560320 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:06.560876 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:06.560881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:06.560885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:06.561571 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:06.561764 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:06.565255 (+ 3491us) server_negotiation.cc:299] Negotiation successful
0927 22:56:06.565671 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4831,"thread_start_us":4594,"threads_started":1}
I20260927 22:56:06.570931 22067 heartbeater.cc:347] Connected to a master server at 127.8.204.190:42333
I20260927 22:56:06.571251 22067 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:06.572119 22067 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
W20260927 22:56:06.574904 22067 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:42333 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:56:06.575117 22067 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:06.575419 22067 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:06.576548 22067 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:06.576810 22067 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
W20260927 22:56:06.577862 22067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:56:06.988389 22071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:06.988777 22071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:06.988977 22071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:07.000749 22071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:07.000897 22071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:07.001137 22071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:56:07.016774 22071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:45311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=41585
--tserver_master_addrs=127.8.204.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:07.018548 22071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:07.020398 22071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:56:07.042821 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:07.042886 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:07.051585 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:07.052464 22071 server_base.cc:1034] running on GCE node
I20260927 22:56:07.053308 22071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:07.054749 22071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:07.056041 22071 hybrid_clock.cc:648] HybridClock initialized: now 1790549767055974 us; error 59 us; skew 500 ppm
I20260927 22:56:07.056469 22071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:07.063851 22071 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:56:07.071749 22071 webserver.cc:460] Webserver started at http://127.8.204.130:41585/ using document root <none> and password file <none>
I20260927 22:56:07.072793 22071 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:07.072969 22071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:07.101020 22071 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 22:56:07.104677 22099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:07.106236 22071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:56:07.106388 22071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "029e074423a942b8ae1c0e5c2a88f7c9"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "31cc57e243e5e9f80e114c6be80f4f16"
server_key_iv: "5e0dd1bc3db4921a1b5386d8496e8d16"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:07.107503 22071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:07.130609 22071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:07.135684 22071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:07.135985 22071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:07.137204 22071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:07.138816 22071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:07.138943 22071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:07.139055 22071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:07.139135 22071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:07.209008 22211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:45311 every 8 connection(s)
I20260927 22:56:07.209283 22071 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:45311
I20260927 22:56:07.210659 22071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:07.215560 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22071
I20260927 22:56:07.216523 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.204.148:60197/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:36093
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:42333
--webserver_port=35689
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:37411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:07.259455 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.219601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:60739 (local address 127.8.204.190:42333)
0927 22:56:07.238436 (+ 18835us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:07.238445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:07.238475 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:56:07.238554 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:07.238562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:07.238670 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:07.238813 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:07.241506 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.242784 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.245997 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.246631 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.246749 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.251239 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:07.251246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:07.251251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:07.251314 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:07.255370 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.256324 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.256330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.256334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.256471 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.257004 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.257009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.257012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.257354 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:07.257495 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:07.258785 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0927 22:56:07.259047 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18661,"thread_start_us":19332,"threads_started":1}
I20260927 22:56:07.260319 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.214648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42333 (local address 127.8.204.130:60739)
0927 22:56:07.227387 (+ 12739us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.227445 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:07.230217 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:07.239497 (+ 9280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:07.239514 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:07.240273 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:07.241243 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:07.241276 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.242994 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.243003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:07.245770 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:07.245797 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.247091 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.247098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:07.247406 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.248241 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:07.248275 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:07.250968 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:07.255541 (+ 4573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:07.255551 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:07.255571 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:07.256176 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:07.256595 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:07.256600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:07.256604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:07.256845 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:07.257563 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:07.257572 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:07.258035 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 22:56:07.258345 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 22:56:07.258944 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12379,"thread_start_us":15832,"threads_started":1}
I20260927 22:56:07.262514 22212 heartbeater.cc:347] Connected to a master server at 127.8.204.190:42333
I20260927 22:56:07.262852 22212 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:07.263819 22212 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:07.266103 21870 ts_manager.cc:194] Registered new tserver with Master: 029e074423a942b8ae1c0e5c2a88f7c9 (127.8.204.130:45311)
I20260927 22:56:07.268275 21870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:60739
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:07.578022 22067 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:07.578492 22067 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:07.579878 21870 ts_manager.cc:194] Registered new tserver with Master: ede4c749c18846048519f3355db69abb (127.8.204.129:44897)
I20260927 22:56:07.581209 21870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:49469
W20260927 22:56:07.621266 22216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:07.621712 22216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:07.621918 22216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:07.633199 22216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:07.633380 22216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:07.633617 22216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:56:07.646348 22216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:37411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.204.148:60197/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:36093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=35689
--tserver_master_addrs=127.8.204.190:42333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:07.647763 22216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:07.649225 22216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:07.666805 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:07.666841 22222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:07.668493 22224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:07.670104 22216 server_base.cc:1034] running on GCE node
I20260927 22:56:07.670848 22216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:07.672101 22216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:07.673358 22216 hybrid_clock.cc:648] HybridClock initialized: now 1790549767673258 us; error 84 us; skew 500 ppm
I20260927 22:56:07.673712 22216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:07.677673 22216 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:56:07.679566 22216 webserver.cc:460] Webserver started at http://127.8.204.131:35689/ using document root <none> and password file <none>
I20260927 22:56:07.680419 22216 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:07.680567 22216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:07.707327 22216 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:56:07.710716 22232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:07.712493 22216 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260927 22:56:07.712635 22216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "75ed027212fa40c488cf8069e5e9d68c"
format_stamp: "Formatted at 2026-09-27 22:56:04 on dist-test-slave-40vx"
server_key: "08c80dbf65446166d9bd2fc1bcd3faf1"
server_key_iv: "efee79a947e2f2cf2ed41d56e2a58c1a"
server_key_version: "kuduclusterkey@0"
I20260927 22:56:07.713836 22216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:07.748623 22216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:07.753669 22216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:07.753971 22216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:07.755179 22216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:07.757314 22216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:07.757409 22216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:07.757505 22216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:07.757563 22216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for HTTP/127.8.204.148@KRBTEST.COM
I20260927 22:56:07.834822 22216 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36093
I20260927 22:56:07.834872 22345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36093 every 8 connection(s)
I20260927 22:56:07.836584 22216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:07.842360 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22216
Sep 27 22:56:07 dist-test-slave-40vx krb5kdc[20859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549767, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:07.889192 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.843407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.8.204.190:42333)
0927 22:56:07.865276 (+ 21869us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:07.865285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:07.865314 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:56:07.865416 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:07.865424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:07.865549 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:07.865689 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:07.868742 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.870014 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.872075 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.872794 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.872912 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.879726 (+ 6814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:07.879732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:07.879737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:07.879778 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:07.884676 (+ 4898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.885559 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.885565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.885569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.885719 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.886206 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.886211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.886214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.886556 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:07.886993 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:07.888592 (+ 1599us) server_negotiation.cc:299] Negotiation successful
0927 22:56:07.888810 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21684,"thread_start_us":22214,"threads_started":1}
I20260927 22:56:07.907191 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.870269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:59253 (local address 127.8.204.190:42333)
0927 22:56:07.873998 (+ 3729us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:07.874006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:07.874028 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:56:07.874095 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:07.874099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:07.874171 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:07.874278 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:07.876501 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.877732 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.880114 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.880796 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.880937 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.890982 (+ 10045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:07.890989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:07.890993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:07.891029 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:07.895476 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.896658 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.896663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.896666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.896804 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.900265 (+ 3461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.900270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.900274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.900620 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:07.900760 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:07.906527 (+ 5767us) server_negotiation.cc:299] Negotiation successful
0927 22:56:07.906785 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3581,"thread_start_us":3447,"threads_started":1}
I20260927 22:56:07.909857 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.869644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:42333 (local address 127.8.204.131:59253)
0927 22:56:07.870754 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:07.870804 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:07.873246 (+ 2442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:07.874807 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:07.874820 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:07.875432 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:07.876279 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:07.876304 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.877913 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.877918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:07.879970 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:07.879984 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.881071 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.881080 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:07.881386 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.886744 (+ 5358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:07.886787 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:07.890692 (+ 3905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:07.895625 (+ 4933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:07.895635 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:07.895654 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:07.896518 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:07.896930 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:07.896935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:07.896938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:07.900107 (+ 3169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:07.900829 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:07.900837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:07.904156 (+ 3319us) client_negotiation.cc:769] Sending connection context
0927 22:56:07.907728 (+ 3572us) client_negotiation.cc:240] Negotiation successful
0927 22:56:07.908390 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":649,"threads_started":1}
I20260927 22:56:07.911697 22346 heartbeater.cc:347] Connected to a master server at 127.8.204.190:42333
I20260927 22:56:07.911988 22346 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:07.912817 22346 heartbeater.cc:511] Master 127.8.204.190:42333 requested a full tablet report, sending...
I20260927 22:56:07.914516 21870 ts_manager.cc:194] Registered new tserver with Master: 75ed027212fa40c488cf8069e5e9d68c (127.8.204.131:36093)
I20260927 22:56:07.915715 21870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:59253
I20260927 22:56:07.920159 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:07.985491 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:07.950350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.8.204.190:42333)
0927 22:56:07.950554 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:07.950561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:07.957923 (+ 7362us) server_negotiation.cc:407] Connection header received
0927 22:56:07.958520 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:07.958527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:07.958619 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:07.958695 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:07.961218 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.962274 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.964526 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:07.965280 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:07.965404 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:07.978620 (+ 13216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:07.978628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:07.978633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:07.978678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:07.982223 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.983166 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.983172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.983175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.983356 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:07.983809 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:07.983814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:07.983817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:07.984160 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:07.984359 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:07.984934 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 22:56:07.985163 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:56:07.992270 21870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:08.003964 21870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:08.081313 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.042209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:45311 (local address 127.0.0.1:41102)
0927 22:56:08.044682 (+ 2473us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.044732 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:08.044899 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:08.070616 (+ 25717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:08.070625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:08.070663 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:08.072650 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.072672 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.075290 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.075296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.077730 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.077745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.079766 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.079773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.080493 (+ 720us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.080547 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 22:56:08.080664 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:56:08.080860 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2220,"mutex_wait_us":479,"thread_start_us":2073,"threads_started":1}
I20260927 22:56:08.085475 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.042739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.8.204.130:45311)
0927 22:56:08.069562 (+ 26823us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.069571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.069610 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:56:08.069734 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.069743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.070067 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.072891 (+ 2824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:08.073110 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.074680 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.077971 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.079523 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.079934 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.080148 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.084967 (+ 4819us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.085152 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26644,"thread_start_us":27533,"threads_started":1}
I20260927 22:56:08.089632 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.046838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36093 (local address 127.0.0.1:32808)
0927 22:56:08.073904 (+ 27066us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.073927 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:08.074033 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:08.076257 (+ 2224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:08.076261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:08.076289 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:08.076701 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.076713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.083356 (+ 6643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.083361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.086514 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.086528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.088900 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.088906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.089111 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.089143 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:56:08.089216 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 22:56:08.089335 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26912,"thread_start_us":27923,"threads_started":1}
I20260927 22:56:08.089658 22147 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 (DEFAULT_TABLE table=test-table [id=c3464c37777843cfa72592ddd23ede33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:08.091154 22147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:08.090737 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.047589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.8.204.131:36093)
0927 22:56:08.074539 (+ 26950us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.074547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.074578 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:56:08.074665 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.074673 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.074910 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.075080 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:08.078547 (+ 3467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.083125 (+ 4578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.086889 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.088412 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.088540 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.088745 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.090164 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.090388 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26757,"thread_start_us":28429,"threads_started":1}
I20260927 22:56:08.095561 22281 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 (DEFAULT_TABLE table=test-table [id=c3464c37777843cfa72592ddd23ede33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:08.097095 22281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:08.102140 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.046463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34322)
0927 22:56:08.058300 (+ 11837us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.058326 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:08.058496 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:08.079721 (+ 21225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:08.079727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:08.079764 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:08.080157 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.080168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.097237 (+ 17069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.097245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.099195 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.099210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.101280 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.101288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.101470 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.101523 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:56:08.101598 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:56:08.101754 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11683,"mutex_wait_us":24,"thread_start_us":11547,"threads_started":1}
I20260927 22:56:08.103659 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.047090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34322 (local address 127.8.204.129:44897)
0927 22:56:08.077382 (+ 30292us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.077392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.077423 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:56:08.077525 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.077533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.077798 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.078044 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:08.095413 (+ 17369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.097024 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.099447 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.101017 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.102831 (+ 1814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.103054 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.103135 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.103314 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30098,"thread_start_us":39524,"threads_started":1}
I20260927 22:56:08.109028 22002 tablet_service.cc:1467] Processing CreateTablet for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 (DEFAULT_TABLE table=test-table [id=c3464c37777843cfa72592ddd23ede33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:08.110507 22002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:08.119294 22367 tablet_bootstrap.cc:492] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Bootstrap starting.
I20260927 22:56:08.122743 22368 tablet_bootstrap.cc:492] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Bootstrap starting.
I20260927 22:56:08.125064 22367 tablet_bootstrap.cc:654] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:08.126361 22367 log.cc:824] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:08.127413 22368 tablet_bootstrap.cc:654] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:08.128654 22368 log.cc:824] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:08.136173 22367 tablet_bootstrap.cc:492] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: No bootstrap required, opened a new log
I20260927 22:56:08.136471 22367 ts_tablet_manager.cc:1397] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20260927 22:56:08.139052 22368 tablet_bootstrap.cc:492] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: No bootstrap required, opened a new log
I20260927 22:56:08.139530 22368 ts_tablet_manager.cc:1397] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260927 22:56:08.147967 22367 raft_consensus.cc:348] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.148406 22367 raft_consensus.cc:374] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:08.148505 22367 raft_consensus.cc:729] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029e074423a942b8ae1c0e5c2a88f7c9, State: Initialized, Role: FOLLOWER
I20260927 22:56:08.149427 22371 tablet_bootstrap.cc:492] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: Bootstrap starting.
I20260927 22:56:08.150928 22368 raft_consensus.cc:348] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.151415 22368 raft_consensus.cc:374] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:08.151540 22368 raft_consensus.cc:729] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ed027212fa40c488cf8069e5e9d68c, State: Initialized, Role: FOLLOWER
I20260927 22:56:08.152737 22368 consensus_queue.cc:260] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.152944 22371 tablet_bootstrap.cc:654] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:08.154178 22371 log.cc:824] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:08.155030 22367 consensus_queue.cc:260] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.161041 22368 ts_tablet_manager.cc:1428] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Time spent starting tablet: real 0.021s user 0.011s sys 0.006s
I20260927 22:56:08.161494 22346 heartbeater.cc:503] Master 127.8.204.190:42333 was elected leader, sending a full tablet report...
I20260927 22:56:08.164995 22212 heartbeater.cc:503] Master 127.8.204.190:42333 was elected leader, sending a full tablet report...
I20260927 22:56:08.165913 22367 ts_tablet_manager.cc:1428] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Time spent starting tablet: real 0.028s user 0.010s sys 0.006s
I20260927 22:56:08.169085 22371 tablet_bootstrap.cc:492] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: No bootstrap required, opened a new log
I20260927 22:56:08.169373 22371 ts_tablet_manager.cc:1397] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20260927 22:56:08.179877 22371 raft_consensus.cc:348] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.180341 22371 raft_consensus.cc:374] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:08.180452 22371 raft_consensus.cc:729] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede4c749c18846048519f3355db69abb, State: Initialized, Role: FOLLOWER
I20260927 22:56:08.181188 22371 consensus_queue.cc:260] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.184005 22067 heartbeater.cc:503] Master 127.8.204.190:42333 was elected leader, sending a full tablet report...
I20260927 22:56:08.184700 22371 ts_tablet_manager.cc:1428] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260927 22:56:08.221860 22213 tablet.cc:2369] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:08.228451 22374 raft_consensus.cc:482] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:08.228724 22374 raft_consensus.cc:504] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.230399 22374 leader_election.cc:288] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ede4c749c18846048519f3355db69abb (127.8.204.129:44897), 75ed027212fa40c488cf8069e5e9d68c (127.8.204.131:36093)
I20260927 22:56:08.238509 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.231086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:37505 (local address 127.8.204.129:44897)
0927 22:56:08.231343 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.231349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.232759 (+ 1410us) server_negotiation.cc:407] Connection header received
0927 22:56:08.232969 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.232978 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.233058 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.233149 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:08.233663 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.234531 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.236361 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.237536 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.237700 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.237794 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.238129 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.238264 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:08.238715 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.230901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:39365 (local address 127.8.204.131:36093)
0927 22:56:08.231097 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.231103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.231930 (+ 827us) server_negotiation.cc:407] Connection header received
0927 22:56:08.232214 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.232219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.232291 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.232348 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:08.233102 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.234074 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.236524 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.237694 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.237831 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.237927 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.238346 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.238480 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:56:08.238548 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.231476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.8.204.130:37505)
0927 22:56:08.232642 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.232659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:08.232822 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:08.233171 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:08.233175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:08.233199 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:08.233499 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.233508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.234658 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.234668 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.236214 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.236225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.237759 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.237764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.237934 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.237970 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:56:08.238070 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:56:08.238189 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":54,"thread_start_us":902,"threads_started":1}
I20260927 22:56:08.239696 22383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.230644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36093 (local address 127.8.204.130:39365)
0927 22:56:08.231822 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.231841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:08.232049 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:08.232550 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:08.232555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:08.232584 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:08.232944 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.232953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.234210 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.234215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.236367 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:08.236380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.237969 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.237975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:08.238163 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.238194 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:56:08.239098 (+ 904us) client_negotiation.cc:240] Negotiation successful
0927 22:56:08.239237 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1050,"mutex_wait_us":6,"thread_start_us":856,"threads_started":1}
I20260927 22:56:08.239981 22022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f7fa38b0495cbc75b5dc5d87a593" candidate_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede4c749c18846048519f3355db69abb" is_pre_election: true
I20260927 22:56:08.240545 22022 raft_consensus.cc:2454] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 029e074423a942b8ae1c0e5c2a88f7c9 in term 0.
I20260927 22:56:08.240635 22301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f7fa38b0495cbc75b5dc5d87a593" candidate_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ed027212fa40c488cf8069e5e9d68c" is_pre_election: true
I20260927 22:56:08.241226 22101 leader_election.cc:302] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029e074423a942b8ae1c0e5c2a88f7c9, ede4c749c18846048519f3355db69abb; no voters:
I20260927 22:56:08.241606 22301 raft_consensus.cc:2454] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 029e074423a942b8ae1c0e5c2a88f7c9 in term 0.
I20260927 22:56:08.241686 22374 raft_consensus.cc:2781] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:08.241824 22374 raft_consensus.cc:482] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:08.241930 22374 raft_consensus.cc:3037] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:08.244275 22374 raft_consensus.cc:504] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.244897 22374 leader_election.cc:288] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [CANDIDATE]: Term 1 election: Requested vote from peers ede4c749c18846048519f3355db69abb (127.8.204.129:44897), 75ed027212fa40c488cf8069e5e9d68c (127.8.204.131:36093)
I20260927 22:56:08.245358 22301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f7fa38b0495cbc75b5dc5d87a593" candidate_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75ed027212fa40c488cf8069e5e9d68c"
I20260927 22:56:08.245368 22022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0aa8f7fa38b0495cbc75b5dc5d87a593" candidate_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede4c749c18846048519f3355db69abb"
I20260927 22:56:08.245571 22301 raft_consensus.cc:3037] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:08.245571 22022 raft_consensus.cc:3037] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:08.248797 22022 raft_consensus.cc:2454] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 029e074423a942b8ae1c0e5c2a88f7c9 in term 1.
I20260927 22:56:08.248797 22301 raft_consensus.cc:2454] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 029e074423a942b8ae1c0e5c2a88f7c9 in term 1.
I20260927 22:56:08.249200 22101 leader_election.cc:302] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 029e074423a942b8ae1c0e5c2a88f7c9, ede4c749c18846048519f3355db69abb; no voters:
I20260927 22:56:08.249501 22374 raft_consensus.cc:2781] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:08.250025 22374 raft_consensus.cc:686] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 1 LEADER]: Becoming Leader. State: Replica: 029e074423a942b8ae1c0e5c2a88f7c9, State: Running, Role: LEADER
I20260927 22:56:08.250556 22374 consensus_queue.cc:237] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } }
I20260927 22:56:08.255507 21869 catalog_manager.cc:5495] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 029e074423a942b8ae1c0e5c2a88f7c9 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "029e074423a942b8ae1c0e5c2a88f7c9" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 } health_report { overall_health: UNKNOWN } } }
I20260927 22:56:08.271885 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.266022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.8.204.130:45311)
0927 22:56:08.266197 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.266204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.266224 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:56:08.266383 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.266388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.266484 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.266628 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:08.266945 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.267986 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.269447 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.270030 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.270186 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.270664 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:08.271266 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:08.271468 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.271557 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.271634 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:56:08.281500 22301 raft_consensus.cc:1261] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 1 FOLLOWER]: Refusing update from remote peer 029e074423a942b8ae1c0e5c2a88f7c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:08.281694 22022 raft_consensus.cc:1261] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb [term 1 FOLLOWER]: Refusing update from remote peer 029e074423a942b8ae1c0e5c2a88f7c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 22:56:08.282343 22068 tablet.cc:2369] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:08.282648 22385 consensus_queue.cc:1035] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ede4c749c18846048519f3355db69abb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:08.283095 22374 consensus_queue.cc:1035] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75ed027212fa40c488cf8069e5e9d68c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:08.298414 22388 mvcc.cc:205] Tried to move back new op lower bound from 7334091850870386688 to 7334091850758471680. Current Snapshot: MvccSnapshot[applied={T|T < 7334091850870386688}]
I20260927 22:56:08.323210 21869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:08.323954 21869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:08.328943 21869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49cce94fa47d48df83027658a1a31d47: Sending DeleteTablet for 3 replicas of tablet 0aa8f7fa38b0495cbc75b5dc5d87a593
I20260927 22:56:08.330307 22147 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:08 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:41102
I20260927 22:56:08.330295 22002 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:08 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34322
I20260927 22:56:08.330991 22281 tablet_service.cc:1514] Processing DeleteTablet for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:08 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:32808
I20260927 22:56:08.340291 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 21907
I20260927 22:56:08.361161 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.341432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.8.204.129:44897)
0927 22:56:08.341626 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.341645 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.341665 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:56:08.341747 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.341755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.341855 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.341968 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:08.344343 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.345753 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.347034 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.347844 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.347971 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.351499 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:08.351536 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:08.351558 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:08.351610 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:08.356474 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:08.357443 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:08.357456 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:08.357462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:08.357640 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:08.358135 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:08.358142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:08.358154 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:08.358650 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:08.358987 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.360390 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.360656 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:56:08.362201 22062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:08.370108 22400 tablet_replica.cc:330] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: stopping tablet replica
I20260927 22:56:08.372288 22400 raft_consensus.cc:2229] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:08.372972 22400 raft_consensus.cc:2258] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:08.379227 22400 ts_tablet_manager.cc:1905] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:08.382407 22398 tablet_replica.cc:330] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: stopping tablet replica
I20260927 22:56:08.383155 22398 raft_consensus.cc:2229] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:08.384240 22398 raft_consensus.cc:2258] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:08.392158 22398 ts_tablet_manager.cc:1905] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:08.404037 22398 ts_tablet_manager.cc:1918] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:08.404232 22398 log.cc:1196] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/0aa8f7fa38b0495cbc75b5dc5d87a593
I20260927 22:56:08.404837 22398 ts_tablet_manager.cc:1939] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 029e074423a942b8ae1c0e5c2a88f7c9: Deleting consensus metadata
I20260927 22:56:08.406908 21855 catalog_manager.cc:4841] TS 029e074423a942b8ae1c0e5c2a88f7c9 (127.8.204.130:45311): tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 (table test-table [id=c3464c37777843cfa72592ddd23ede33]) successfully deleted
I20260927 22:56:08.409870 22400 ts_tablet_manager.cc:1918] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:08.410053 22400 log.cc:1196] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/0aa8f7fa38b0495cbc75b5dc5d87a593
I20260927 22:56:08.410648 22400 ts_tablet_manager.cc:1939] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P 75ed027212fa40c488cf8069e5e9d68c: Deleting consensus metadata
I20260927 22:56:08.412947 21856 catalog_manager.cc:4841] TS 75ed027212fa40c488cf8069e5e9d68c (127.8.204.131:36093): tablet 0aa8f7fa38b0495cbc75b5dc5d87a593 (table test-table [id=c3464c37777843cfa72592ddd23ede33]) successfully deleted
I20260927 22:56:08.489943 22399 tablet_replica.cc:330] T 0aa8f7fa38b0495cbc75b5dc5d87a593 P ede4c749c18846048519f3355db69abb: stopping tablet replica
I20260927 22:56:08.490227 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 21907
I20260927 22:56:08.506973 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22071
W20260927 22:56:08.506999 21855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.129:44897 (error 108)
I20260927 22:56:08.509127 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.508590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34336)
0927 22:56:08.508823 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.508949 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260927 22:56:08.509398 21855 catalog_manager.cc:4584] TS ede4c749c18846048519f3355db69abb (127.8.204.129:44897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0aa8f7fa38b0495cbc75b5dc5d87a593: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
I20260927 22:56:08.526958 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.508173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.8.204.130:45311)
0927 22:56:08.508344 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.508348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.508366 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:08.508421 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.508424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.508485 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.508596 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:08.510074 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.510925 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.512506 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.512918 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.513064 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.520154 (+ 7090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:08.520170 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:08.520191 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:08.520242 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:08.524210 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:08.524871 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:08.524880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:08.524883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:08.525048 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:08.525390 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:08.525395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:08.525399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:08.525837 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:08.526172 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.526324 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.526547 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:56:08.527814 22207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:08.563086 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.562573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34350)
0927 22:56:08.562799 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.562923 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:56:08.564074 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.563767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34360)
0927 22:56:08.563890 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.563972 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:56:08.624766 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.624293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34372)
0927 22:56:08.624521 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.624615 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:56:08.625869 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.625567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34382)
0927 22:56:08.625698 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.625765 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:56:08.643366 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22071
I20260927 22:56:08.661460 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22216
I20260927 22:56:08.674434 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.662180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.8.204.131:36093)
0927 22:56:08.662331 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:08.662335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:08.662353 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:08.662503 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:08.662507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:08.662576 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:08.662660 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:08.663940 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.664682 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.665917 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:08.666318 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:08.666466 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:08.668976 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:08.668990 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:08.669005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:08.669040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:08.671801 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:08.672449 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:08.672456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:08.672458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:08.672586 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:08.672962 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:08.672965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:08.672967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:08.673294 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:08.673597 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:08.673887 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:56:08.674111 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:56:08.675096 22341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:08.706451 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.705971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34386)
0927 22:56:08.706175 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.706268 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:56:08.707592 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.707170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34396)
0927 22:56:08.707408 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.707482 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:56:08.762425 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22216
I20260927 22:56:08.778918 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 21837
I20260927 22:56:08.780181 21900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:08.893128 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:08.892244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44897 (local address 127.0.0.1:34402)
0927 22:56:08.892442 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:08.892575 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:44897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:56:08.893551 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 21837
I20260927 22:56:08.915237 9010 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:56:09.250088 9010 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:56:09.250275 9010 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:56:09.595223 9010 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:56:09.595 UTC [20889] LOG: received smart shutdown request
2026-09-27 22:56:09.598 UTC [20889] LOG: background worker "logical replication launcher" (PID 20898) exited with exit code 1
2026-09-27 22:56:09.598 UTC [20893] LOG: shutting down
2026-09-27 22:56:09.757 UTC [20889] LOG: database system is shut down
2026-09-27T22:56:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52535 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:09 dist-test-slave-40vx krb5kdc[22414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:09 dist-test-slave-40vx krb5kdc[22414](info): set up 2 sockets
Sep 27 22:56:09 dist-test-slave-40vx krb5kdc[22414](info): commencing operation
krb5kdc: starting...
W20260927 22:56:10.971637 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:56:11 dist-test-slave-40vx krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:56:11.191191 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:33695
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:33695,127.8.204.189:42909,127.8.204.188:45295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:11.536496 22427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:11.536885 22427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:11.537051 22427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:11.544456 22427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:11.544584 22427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:11.544685 22427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:11.544775 22427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:11.544853 22427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:11.554832 22427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:33695,127.8.204.189:42909,127.8.204.188:45295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:33695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:11.556242 22427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:11.557564 22427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:11.569517 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:11.569554 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:11.569913 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:11.570573 22427 server_base.cc:1034] running on GCE node
I20260927 22:56:11.571455 22427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:11.572885 22427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:11.574141 22427 hybrid_clock.cc:648] HybridClock initialized: now 1790549771574101 us; error 59 us; skew 500 ppm
I20260927 22:56:11.574501 22427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:11 dist-test-slave-40vx krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:11.577744 22427 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:11.579453 22427 webserver.cc:460] Webserver started at http://127.8.204.190:42309/ using document root <none> and password file <none>
I20260927 22:56:11.580173 22427 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:11.580331 22427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:11.580631 22427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:11.583894 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "17a820419aca46f5a0e1ef758ef15bc1"
format_stamp: "Formatted at 2026-09-27 22:56:11 on dist-test-slave-40vx"
server_key: "382d911587ac096972cdaacdc7a9d4f3"
server_key_iv: "53c592d53775e85f2efe7e7f212f1b6d"
server_key_version: "encryptionkey@0"
I20260927 22:56:11.584717 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "17a820419aca46f5a0e1ef758ef15bc1"
format_stamp: "Formatted at 2026-09-27 22:56:11 on dist-test-slave-40vx"
server_key: "382d911587ac096972cdaacdc7a9d4f3"
server_key_iv: "53c592d53775e85f2efe7e7f212f1b6d"
server_key_version: "encryptionkey@0"
I20260927 22:56:11.590075 22427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:56:11.593585 22442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:11.594945 22427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:56:11.595127 22427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "17a820419aca46f5a0e1ef758ef15bc1"
format_stamp: "Formatted at 2026-09-27 22:56:11 on dist-test-slave-40vx"
server_key: "382d911587ac096972cdaacdc7a9d4f3"
server_key_iv: "53c592d53775e85f2efe7e7f212f1b6d"
server_key_version: "encryptionkey@0"
I20260927 22:56:11.595378 22427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:11.621896 22427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:11.626371 22427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:11.626667 22427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:11.645455 22427 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:33695
I20260927 22:56:11.645478 22493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:33695 every 8 connection(s)
I20260927 22:56:11.646914 22427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:11.650735 22494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:11.651834 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22427
I20260927 22:56:11.652070 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:56:11.652717 9010 external_mini_cluster.cc:1428] Setting key 1207bb3fad86234358e780e7ed83fed9
I20260927 22:56:11.657393 22494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.204.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.189@KRBTEST.COM" created.
Sep 27 22:56:11 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:11.685742 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.660779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33695 (local address 127.0.0.1:35650)
0927 22:56:11.661661 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.661715 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:11.664618 (+ 2903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:11.665505 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:11.665512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:11.666371 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:11.666888 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:11.666917 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.668585 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.668594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:11.670608 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:11.670628 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.671663 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.671673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:11.671863 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.672616 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:11.672650 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:11.678202 (+ 5552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:11.681407 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:11.681422 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:11.681437 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:11.682015 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:11.682390 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:11.682397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:11.682403 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:11.682621 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:11.683519 (+ 898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:11.683531 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:11.683803 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 22:56:11.684137 (+ 334us) client_negotiation.cc:240] Negotiation successful
0927 22:56:11.684739 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":147,"thread_start_us":405,"threads_started":1}
I20260927 22:56:11.685791 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.660779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.8.204.190:33695)
0927 22:56:11.662684 (+ 1905us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:11.662690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:11.662732 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:56:11.664995 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:11.665001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:11.665252 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:11.665409 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:11.667102 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.668419 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.670758 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:11.671408 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:11.671544 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:11.678442 (+ 6898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:11.678449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:11.678462 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:11.678491 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:11.681259 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.682131 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.682136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.682138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.682276 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:11.682738 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:11.682741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:11.682743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:11.683235 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:11.683495 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:11.684240 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 22:56:11.684607 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1731,"spinlock_wait_cycles":55552,"thread_start_us":1510,"threads_started":1}
I20260927 22:56:11.688800 22494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:11.689942 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.689561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:39956)
0927 22:56:11.689742 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.689839 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 22:56:11.690268 22446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
I20260927 22:56:11.691751 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.691445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:39972)
0927 22:56:11.691565 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.691633 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:56:11.692028 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42909: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab.
Entry for principal kudu/127.8.204.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab.
I20260927 22:56:11.704728 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:42909
--webserver_interface=127.8.204.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab
--principal=kudu/127.8.204.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:33695,127.8.204.189:42909,127.8.204.188:45295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:11.741434 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } attempt: 1
I20260927 22:56:11.742638 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.742200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:39984)
0927 22:56:11.742378 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.742499 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:56:11.743728 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.743399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:39996)
0927 22:56:11.743525 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.743589 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 22:56:11.743935 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42909: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
I20260927 22:56:11.812197 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } attempt: 2
I20260927 22:56:11.813652 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.813061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40006)
0927 22:56:11.813344 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.813465 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:56:11.815192 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.814589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40020)
0927 22:56:11.814935 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.815029 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260927 22:56:11.815456 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42909: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
I20260927 22:56:11.906788 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } attempt: 3
I20260927 22:56:11.908008 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.907582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40030)
0927 22:56:11.907772 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.907880 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:56:11.909067 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:11.908696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40032)
0927 22:56:11.908843 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:11.908910 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:56:11.909291 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42909: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
W20260927 22:56:12.027333 22501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:12.027623 22501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:12.027801 22501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:12.034956 22501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:12.035069 22501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:12.035185 22501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:12.035315 22501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:12.035403 22501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:12.045488 22501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:33695,127.8.204.189:42909,127.8.204.188:45295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab
--principal=kudu/127.8.204.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:42909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:12.046811 22501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:12.048152 22501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:56:12.052574 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } attempt: 4
I20260927 22:56:12.054018 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.053295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40038)
0927 22:56:12.053698 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.053813 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260927 22:56:12.055366 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.054978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40046)
0927 22:56:12.055117 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.055199 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:56:12.055558 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42909: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42909: connect: Connection refused (error 111)
W20260927 22:56:12.060770 22507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.060746 22509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.060833 22506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:12.061738 22501 server_base.cc:1034] running on GCE node
I20260927 22:56:12.062508 22501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:12.063858 22501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:12.065052 22501 hybrid_clock.cc:648] HybridClock initialized: now 1790549772065014 us; error 28 us; skew 500 ppm
I20260927 22:56:12.065351 22501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:12.071261 22501 init.cc:377] Logged in from keytab as kudu/127.8.204.189@KRBTEST.COM (short username kudu)
I20260927 22:56:12.072688 22501 webserver.cc:460] Webserver started at http://127.8.204.189:33405/ using document root <none> and password file <none>
I20260927 22:56:12.073349 22501 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:12.073491 22501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:12.073777 22501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:12.076710 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data/instance:
uuid: "db33b617baa64586b78d358b8295044e"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-40vx"
server_key: "0fe523d7ddaefa0066f3e766a75c9bb1"
server_key_iv: "7e7df3bb1ca22dc29acfe178b729c9ca"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.077512 22501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal/instance:
uuid: "db33b617baa64586b78d358b8295044e"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-40vx"
server_key: "0fe523d7ddaefa0066f3e766a75c9bb1"
server_key_iv: "7e7df3bb1ca22dc29acfe178b729c9ca"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.082353 22501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:56:12.085659 22516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:12.086869 22501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:56:12.087069 22501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal
uuid: "db33b617baa64586b78d358b8295044e"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-40vx"
server_key: "0fe523d7ddaefa0066f3e766a75c9bb1"
server_key_iv: "7e7df3bb1ca22dc29acfe178b729c9ca"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.087265 22501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:12.116824 22501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:12.121274 22501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:12.121567 22501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:12.139976 22501 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.189:42909
I20260927 22:56:12.140005 22567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.189:42909 every 8 connection(s)
I20260927 22:56:12.141340 22501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
I20260927 22:56:12.144863 22568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:12.146102 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22501
I20260927 22:56:12.146294 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-1/wal/instance
I20260927 22:56:12.146708 9010 external_mini_cluster.cc:1428] Setting key 25cf09fdf784d02a4cd9cd4c8d76b19b
I20260927 22:56:12.151376 22568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.188@KRBTEST.COM" created.
I20260927 22:56:12.172217 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.153871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.8.204.190:33695)
0927 22:56:12.154098 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.154106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.155353 (+ 1247us) server_negotiation.cc:407] Connection header received
0927 22:56:12.157142 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.157150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.157240 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.157370 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.158888 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.160088 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.161659 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.162399 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.162565 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.165899 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.165908 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.165916 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.165960 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.169055 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.169988 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.169991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.169994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.170104 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.170545 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.170549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.170551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.170805 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.170921 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.171837 (+ 916us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.171999 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:56:12.173892 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.154364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33695 (local address 127.0.0.1:35658)
0927 22:56:12.155173 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.155217 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.156888 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.157569 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.157577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.158148 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.158713 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.158730 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.160242 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.160246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.161488 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.161500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.162603 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.162609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.162788 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.163558 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.163579 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.165634 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.169189 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.169201 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.169219 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.169877 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.170205 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.170213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.170219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.170438 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.170939 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.170949 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.171439 (+ 490us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.171763 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.172301 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":332,"threads_started":1}
I20260927 22:56:12.175664 22568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:56:12.191968 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.176475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40058)
0927 22:56:12.176676 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.176696 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.176814 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.177549 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.177561 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.178056 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.178268 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.178281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.179631 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.179639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.181248 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.181264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.182043 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.182056 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.182311 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.182888 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.182915 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.185323 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.189665 (+ 4342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.189668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.189671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.189990 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.190462 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.190465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.190467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.190610 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.191209 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.191214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.191381 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.191494 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.191697 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:56:12.192405 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.176552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.8.204.189:42909)
0927 22:56:12.177160 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.177166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.177190 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:56:12.177231 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.177236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.177395 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.177508 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.178416 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.179504 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.181380 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.181899 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.181991 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.185588 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.185604 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.185622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.185663 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.189531 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.190199 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.190208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.190211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.190353 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.190707 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.190710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.190712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.191091 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.191340 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.191687 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.191910 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":317,"threads_started":1}
I20260927 22:56:12.194661 22568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:12.195895 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.195447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43150)
0927 22:56:12.195602 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.195729 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:56:12.196195 22518 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab.
Entry for principal kudu/127.8.204.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab.
I20260927 22:56:12.197726 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.197366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43154)
0927 22:56:12.197510 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.197590 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:56:12.197943 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.198531 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:45295
--webserver_interface=127.8.204.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab
--principal=kudu/127.8.204.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:33695,127.8.204.189:42909,127.8.204.188:45295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:12.247324 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 1
I20260927 22:56:12.248452 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.248004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43164)
0927 22:56:12.248196 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.248309 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:56:12.249429 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.249090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43174)
0927 22:56:12.249272 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.249332 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 22:56:12.249631 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.317860 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 2
I20260927 22:56:12.318959 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.318488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43186)
0927 22:56:12.318720 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.318808 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:56:12.319957 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.319641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43196)
0927 22:56:12.319778 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.319848 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:56:12.320159 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.354818 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } attempt: 5
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:56:12.369724 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.355733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.8.204.189:42909)
0927 22:56:12.355909 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.355913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.355932 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:56:12.356132 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.356136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.356211 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.356296 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.357256 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.357924 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.359636 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.360244 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.360352 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.364055 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.364061 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.364064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.364098 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.366768 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.367760 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.367764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.367767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.367886 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.368333 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.368336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.368338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.368609 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.368723 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.369272 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.369439 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:12.369908 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.355612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40066)
0927 22:56:12.355840 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.355864 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.356002 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.356365 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.356370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.356880 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.357116 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.357129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.358052 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.358058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.359482 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.359497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.360407 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.360413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.360604 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.361253 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.361276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.363854 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.366940 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.366948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.366955 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.367635 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.368009 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.368015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.368019 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.368217 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.368741 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.368749 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.368954 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.369186 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.369565 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:56:12.371008 22494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:56:12.372241 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.371802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43204)
0927 22:56:12.371987 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.372082 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:56:12.373199 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.372843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43220)
0927 22:56:12.373002 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.373081 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:56:12.373399 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.411394 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 3
I20260927 22:56:12.412587 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.412133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43222)
0927 22:56:12.412340 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.412434 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 22:56:12.413820 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.413418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43230)
0927 22:56:12.413648 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.413718 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260927 22:56:12.414067 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.424618 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 1
I20260927 22:56:12.425804 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.425239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43242)
0927 22:56:12.425477 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.425596 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:56:12.426865 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.426452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43254)
0927 22:56:12.426616 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.426711 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:56:12.427124 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.495371 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 2
I20260927 22:56:12.496636 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.496053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43256)
0927 22:56:12.496290 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.496421 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:56:12.497704 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.497314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43266)
0927 22:56:12.497472 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.497560 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 22:56:12.497961 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
W20260927 22:56:12.527055 22575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:12.527366 22575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:12.527477 22575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:12.534360 22575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:12.534480 22575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:12.534524 22575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:12.534571 22575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:12.534615 22575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:12.543864 22575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:33695,127.8.204.189:42909,127.8.204.188:45295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab
--principal=kudu/127.8.204.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:45295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:12.545002 22575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:12.546154 22575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:12.557478 22580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:12.557427 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 4
W20260927 22:56:12.557442 22583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:12.557873 22581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:12.558581 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.558073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43282)
0927 22:56:12.558283 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.558403 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:56:12.559031 22575 server_base.cc:1034] running on GCE node
I20260927 22:56:12.559592 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.559229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43296)
0927 22:56:12.559382 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.559469 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 22:56:12.559831 22568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.559988 22575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:12.561345 22575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:12.562534 22575 hybrid_clock.cc:648] HybridClock initialized: now 1790549772562490 us; error 39 us; skew 500 ppm
I20260927 22:56:12.562832 22575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:12.565874 22575 init.cc:377] Logged in from keytab as kudu/127.8.204.188@KRBTEST.COM (short username kudu)
I20260927 22:56:12.567323 22575 webserver.cc:460] Webserver started at http://127.8.204.188:39385/ using document root <none> and password file <none>
I20260927 22:56:12.568037 22575 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:12.568162 22575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:12.568470 22575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:12.571460 22575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data/instance:
uuid: "1493efebde2b409a8af4bc675bebd448"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-40vx"
server_key: "e1d1664851d19d34bc972887bb7527bd"
server_key_iv: "4754b509ba15fafc8b433ba587bb921d"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.572255 22575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal/instance:
uuid: "1493efebde2b409a8af4bc675bebd448"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-40vx"
server_key: "e1d1664851d19d34bc972887bb7527bd"
server_key_iv: "4754b509ba15fafc8b433ba587bb921d"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.577203 22575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:12.580470 22590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:12.581784 22575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:12.581955 22575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal
uuid: "1493efebde2b409a8af4bc675bebd448"
format_stamp: "Formatted at 2026-09-27 22:56:12 on dist-test-slave-40vx"
server_key: "e1d1664851d19d34bc972887bb7527bd"
server_key_iv: "4754b509ba15fafc8b433ba587bb921d"
server_key_version: "encryptionkey@0"
I20260927 22:56:12.582144 22575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:12.593962 22575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:12.597826 22575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:12.598094 22575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:12.604171 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 3
I20260927 22:56:12.605329 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.604716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43306)
0927 22:56:12.604970 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.605099 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:56:12.606456 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.606100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43308)
0927 22:56:12.606283 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.606354 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 22:56:12.606711 22494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:45295: Network error: Client connection negotiation failed: client connection to 127.8.204.188:45295: connect: Connection refused (error 111)
I20260927 22:56:12.616259 22575 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.188:45295
I20260927 22:56:12.616282 22641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.188:45295 every 8 connection(s)
I20260927 22:56:12.617547 22575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
I20260927 22:56:12.619048 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22575
I20260927 22:56:12.619270 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790549572925352-9010-0/minicluster-data/master-2/wal/instance
I20260927 22:56:12.619812 9010 external_mini_cluster.cc:1428] Setting key cbfb4c627bfbb71e96bd02ad915f0d97
I20260927 22:56:12.621964 22642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:12.630613 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:12.643028 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.621571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.8.204.188:45295)
0927 22:56:12.622369 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.622384 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.622430 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:56:12.624642 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.624666 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.625180 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.625591 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.626844 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.628882 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.630188 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.631211 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.631452 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.635809 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.635816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.635825 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.635857 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.638671 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.639628 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.639634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.639639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.639801 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.640197 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.640200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.640202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.640690 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.640984 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.641432 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.641803 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":326,"threads_started":1}
I20260927 22:56:12.647921 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.633382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.8.204.190:33695)
0927 22:56:12.633540 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.633544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.634155 (+ 611us) server_negotiation.cc:407] Connection header received
0927 22:56:12.634352 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.634357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.634429 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.634527 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.635647 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.636321 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.637684 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.638121 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.638255 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.641565 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.641569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.641573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.641602 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.644296 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.644949 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.644956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.644963 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.645197 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.645897 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.645905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.645908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.646309 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.646473 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.647312 (+ 839us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.647533 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:56:12.648629 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.633343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33695 (local address 127.0.0.1:35666)
0927 22:56:12.634014 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.634055 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.634231 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.634591 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.634598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.635194 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.635504 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.635520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.636483 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.636486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.637547 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.637561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.638293 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.638299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.638483 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.639070 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.639091 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.641385 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.644423 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.644433 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.644440 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.644839 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.645619 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.645622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.645624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.645763 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.646603 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.646613 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.646863 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.647138 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.647537 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"spinlock_wait_cycles":11136,"thread_start_us":369,"threads_started":1}
I20260927 22:56:12.653051 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:56:12.665355 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.645939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.8.204.190:33695)
0927 22:56:12.648121 (+ 2182us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.648126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.648144 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:56:12.648186 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.648191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.648253 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.648339 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.649993 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.651545 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.653375 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.654014 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.654132 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.659352 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.659356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.659359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.659387 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.662500 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.663400 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.663407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.663411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.663568 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.663985 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.663991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.663995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.664359 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.664651 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.664922 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.665080 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2083,"thread_start_us":5485,"threads_started":1}
I20260927 22:56:12.667292 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.653692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42909 (local address 127.0.0.1:40074)
0927 22:56:12.653862 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.653889 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.653985 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.654484 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.654488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.654937 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.655163 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.655179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.656663 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.656667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.657519 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.657529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.658214 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.658223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.658456 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.658992 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.659016 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.660945 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.664651 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.664656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.664660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.665195 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.665598 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.665602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.665605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.665772 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.666313 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.666318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.666485 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.666630 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.666908 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:56:12.667500 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.653966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.8.204.189:42909)
0927 22:56:12.654161 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.654167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.654187 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:56:12.654247 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.654253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.654333 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.654438 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.655382 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.656514 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.657656 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.658060 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.658152 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.661143 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.661150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.661156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.661200 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.664514 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.665315 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.665320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.665323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.665483 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.665888 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.665894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.665897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.666149 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.666309 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.666779 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.667007 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:56:12.668303 22642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:56:12.681548 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.668995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43336)
0927 22:56:12.669180 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.669198 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.669291 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.669551 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.669556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.669901 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.670024 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.670031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.671084 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.671090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.672348 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.672360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.673202 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.673207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.673414 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.673920 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.673936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.676154 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.679159 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.679164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.679168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.679572 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.679929 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.679933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.679937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.680057 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.680540 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.680544 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.680695 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.680818 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.681116 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:56:12.681624 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.669164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.8.204.188:45295)
0927 22:56:12.669308 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.669312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.669345 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:56:12.669380 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.669383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.669430 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.669494 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.670149 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.670958 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.672475 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.673050 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.673186 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.676324 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.676329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.676332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.676359 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.679027 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.679684 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.679691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.679694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.679809 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.680166 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.680171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.680174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.680432 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.680576 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.680903 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.681272 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":45}
I20260927 22:56:12.682636 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.666979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.8.204.189:42909)
0927 22:56:12.668018 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.668023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.668039 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:56:12.668083 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.668093 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.668148 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.668251 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.669632 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.670448 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.672103 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.672597 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.672721 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.677130 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.677135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.677138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.677169 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.679728 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.680372 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.680376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.680378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.680503 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.680899 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.680902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.680905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.681145 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.681425 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.682211 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.682396 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":847,"threads_started":1}
I20260927 22:56:12.687772 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:12.687734 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: Bootstrap starting.
I20260927 22:56:12.691962 22642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:12.693346 22642 log.cc:824] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:12.704777 22642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: No bootstrap required, opened a new log
I20260927 22:56:12.716602 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.694050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.8.204.190:33695)
0927 22:56:12.694295 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.694304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.697884 (+ 3580us) server_negotiation.cc:407] Connection header received
0927 22:56:12.698111 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.698117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.698225 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.698335 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.701652 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.702794 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.705186 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.705816 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.705912 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.709787 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.709793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.709799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.709843 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.713304 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.714244 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.714250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.714254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.714418 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.714858 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.714863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.714867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.715213 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.715476 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.716004 (+ 528us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.716261 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:56:12.716598 22642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.717056 22642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:12.717185 22642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1493efebde2b409a8af4bc675bebd448, State: Initialized, Role: FOLLOWER
I20260927 22:56:12.717989 22642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.719048 22657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.719518 22657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:12.720360 22642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:12.721551 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.694180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.8.204.189:42909)
0927 22:56:12.698135 (+ 3955us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.698144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.702604 (+ 4460us) server_negotiation.cc:407] Connection header received
0927 22:56:12.702813 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.702819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.702899 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.702988 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.705883 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.706950 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.709376 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.710025 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.710121 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.713944 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.713953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.713961 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.714007 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.718202 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.719092 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.719097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.719100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.719304 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.719797 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.719802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.719805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.720158 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.720419 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.720935 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.721199 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3771}
I20260927 22:56:12.724020 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.694478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.8.204.188:45295)
0927 22:56:12.694711 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.694720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.702132 (+ 7412us) server_negotiation.cc:407] Connection header received
0927 22:56:12.702839 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.702844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.702913 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.702978 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.705767 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.706906 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.709376 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.710114 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.710212 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.714298 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.714304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.714308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.714349 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.718960 (+ 4611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.720587 (+ 1627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.720593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.720596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.720748 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.721366 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.721371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.721376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.721797 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.723265 (+ 1468us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.723432 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.723657 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 22:56:12.731501 22668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:12.731642 22668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:12.727982 22575 master_runner.cc:186] Error getting master registration for 127.8.204.190:33695: OK, instance_id { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" instance_seqno: 1790549771639912 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:56:12.735359 22575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:56:12.737033 22657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:12.737272 22657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.738499 22657 leader_election.cc:288] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17a820419aca46f5a0e1ef758ef15bc1 (127.8.204.190:33695), db33b617baa64586b78d358b8295044e (127.8.204.189:42909)
I20260927 22:56:12.783926 22494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 4
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549771, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:56:12.801998 22497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.784560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43346)
0927 22:56:12.785014 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.785035 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.785144 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.786543 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.786548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.787137 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.787384 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.787397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.788875 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.788881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.790393 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.790408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.791400 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.791405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.791614 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.792306 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.792328 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.795130 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.799196 (+ 4066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.799202 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.799206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.799771 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.800192 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.800197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.800201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.800359 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.800977 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.800982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.801194 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.801336 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.801624 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260927 22:56:12.802892 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.785611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.8.204.188:45295)
0927 22:56:12.786017 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.786023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.786040 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:56:12.786090 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.786096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.786181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.786324 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.787530 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.788741 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.790551 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.791211 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.791392 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.795368 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.795377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.795384 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.795420 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.799067 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.799897 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.799903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.799906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.800083 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.800464 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.800470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.800473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.800853 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.800991 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.802328 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.802545 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260927 22:56:12.807520 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: Bootstrap starting.
I20260927 22:56:12.811477 22494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:12.812742 22494 log.cc:824] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:12.816149 22494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: No bootstrap required, opened a new log
I20260927 22:56:12.827674 22494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.828075 22494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:12.828189 22494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a820419aca46f5a0e1ef758ef15bc1, State: Initialized, Role: FOLLOWER
I20260927 22:56:12.828859 22494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.829797 22671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.830224 22671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:12.831175 22494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:12.831246 22469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1493efebde2b409a8af4bc675bebd448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a820419aca46f5a0e1ef758ef15bc1" is_pre_election: true
I20260927 22:56:12.831845 22469 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1493efebde2b409a8af4bc675bebd448 in term 0.
I20260927 22:56:12.832509 22593 leader_election.cc:302] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1493efebde2b409a8af4bc675bebd448, 17a820419aca46f5a0e1ef758ef15bc1; no voters:
I20260927 22:56:12.833066 22657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:12.833233 22657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:12.833345 22657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:12.836690 22657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.837399 22657 leader_election.cc:288] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [CANDIDATE]: Term 1 election: Requested vote from peers 17a820419aca46f5a0e1ef758ef15bc1 (127.8.204.190:33695), db33b617baa64586b78d358b8295044e (127.8.204.189:42909)
I20260927 22:56:12.838542 22469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1493efebde2b409a8af4bc675bebd448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17a820419aca46f5a0e1ef758ef15bc1"
I20260927 22:56:12.839046 22469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:12.842753 22469 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1493efebde2b409a8af4bc675bebd448 in term 1.
I20260927 22:56:12.843194 22593 leader_election.cc:302] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1493efebde2b409a8af4bc675bebd448, 17a820419aca46f5a0e1ef758ef15bc1; no voters:
I20260927 22:56:12.843583 22657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 1 FOLLOWER]: Leader election won for term 1
W20260927 22:56:12.850153 22596 tablet.cc:2369] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:12.854269 22657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [term 1 LEADER]: Becoming Leader. State: Replica: 1493efebde2b409a8af4bc675bebd448, State: Running, Role: LEADER
I20260927 22:56:12.855458 22657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.859045 22568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } attempt: 5
I20260927 22:56:12.860242 22677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1493efebde2b409a8af4bc675bebd448. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.860697 22677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:12.867822 22681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 22:56:12 dist-test-slave-40vx krb5kdc[22414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549772, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:56:12.870302 22681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 22:56:12.879503 22448 tablet.cc:2369] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:12.879729 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.859674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:45295 (local address 127.0.0.1:43354)
0927 22:56:12.860018 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:12.860041 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:12.860154 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:12.861178 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:12.861184 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:12.861735 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:12.861940 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.861953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.863456 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.863462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.864992 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:12.865008 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.866205 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.866211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:12.866468 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.867197 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:12.867221 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:12.870149 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:12.874331 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.874336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.874368 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.874916 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.875501 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:12.875507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:12.875510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:12.875686 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:12.878568 (+ 2882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:12.878573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:12.878798 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 22:56:12.878960 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:56:12.879294 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260927 22:56:12.880273 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:12.860494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43354 (local address 127.8.204.188:45295)
0927 22:56:12.860684 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:12.860690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:12.860707 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:56:12.860763 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:12.860767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:12.860846 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:12.860949 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:12.862119 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.863264 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.865165 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:12.865884 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:12.865999 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:12.870364 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:12.870372 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:12.870378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:12.870422 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:12.874185 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.875046 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.875051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.875088 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.875326 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:12.875806 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:12.875811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:12.875814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:12.878427 (+ 2613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:12.878576 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:12.879146 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 22:56:12.879659 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:56:12.886235 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: Bootstrap starting.
I20260927 22:56:12.890806 22568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:12.892820 22469 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [term 1 FOLLOWER]: Refusing update from remote peer 1493efebde2b409a8af4bc675bebd448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:12.893213 22568 log.cc:824] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:12.894155 22677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:12.905081 22568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: No bootstrap required, opened a new log
I20260927 22:56:12.915649 22671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1493efebde2b409a8af4bc675bebd448. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.915978 22671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:12.919469 22684 mvcc.cc:205] Tried to move back new op lower bound from 7334091869751533568 to 7334091869621035008. Current Snapshot: MvccSnapshot[applied={T|T < 7334091869751533568}]
I20260927 22:56:12.922883 22657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.923254 22657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:12.930363 22568 raft_consensus.cc:348] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.930783 22568 raft_consensus.cc:374] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:12.930898 22568 raft_consensus.cc:729] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db33b617baa64586b78d358b8295044e, State: Initialized, Role: FOLLOWER
I20260927 22:56:12.931595 22671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.931929 22671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:12.932753 22681 catalog_manager.cc:1347] Generated new cluster ID: adac2abd3a694e0fa0e4e2639fba77cf
I20260927 22:56:12.932870 22681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:12.933238 22568 consensus_queue.cc:260] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } }
I20260927 22:56:12.935763 22687 mvcc.cc:205] Tried to move back new op lower bound from 7334091869751533568 to 7334091869621035008. Current Snapshot: MvccSnapshot[applied={T|T < 7334091869751533568 or (T in {7334091869751533568})}]
W20260927 22:56:12.941991 22689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:56:12.943733 22689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:56:12.925549 22427 master_runner.cc:186] Error getting master registration for 127.8.204.189:42909: OK, instance_id { permanent_uuid: "db33b617baa64586b78d358b8295044e" instance_seqno: 1790549772134730 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:56:12.947695 22427 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:56:12.948730 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:12.949561 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:12.949883 22568 sys_catalog.cc:564] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:12.950291 22541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:12.951368 22543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1493efebde2b409a8af4bc675bebd448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33b617baa64586b78d358b8295044e" is_pre_election: true
I20260927 22:56:12.952064 22542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1493efebde2b409a8af4bc675bebd448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db33b617baa64586b78d358b8295044e"
I20260927 22:56:12.955698 22541 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [term 1 FOLLOWER]: Refusing update from remote peer 1493efebde2b409a8af4bc675bebd448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:12.956426 22677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:12.978091 22681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:12.979974 22681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:13.002918 22681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448: Generated new TSK 0
I20260927 22:56:13.004043 22681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:13.005138 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:13.005580 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:13.005987 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1493efebde2b409a8af4bc675bebd448. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:13.006242 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:56:13.007584 22677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1493efebde2b409a8af4bc675bebd448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17a820419aca46f5a0e1ef758ef15bc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33695 } } peers { permanent_uuid: "db33b617baa64586b78d358b8295044e" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42909 } } peers { permanent_uuid: "1493efebde2b409a8af4bc675bebd448" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 45295 } } }
I20260927 22:56:13.007860 22677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1493efebde2b409a8af4bc675bebd448 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:13.064653 22712 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:56:13.946481 22689 catalog_manager.cc:1259] Loaded cluster ID: adac2abd3a694e0fa0e4e2639fba77cf
I20260927 22:56:13.946586 22689 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: loading cluster ID for follower catalog manager: success
I20260927 22:56:13.948992 22689 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: acquiring CA information for follower catalog manager: success
I20260927 22:56:13.950747 22689 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17a820419aca46f5a0e1ef758ef15bc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:56:14.069190 22712 catalog_manager.cc:1259] Loaded cluster ID: adac2abd3a694e0fa0e4e2639fba77cf
I20260927 22:56:14.069295 22712 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: loading cluster ID for follower catalog manager: success
I20260927 22:56:14.071770 22712 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: acquiring CA information for follower catalog manager: success
I20260927 22:56:14.073571 22712 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db33b617baa64586b78d358b8295044e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:56:14.131645 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22427
I20260927 22:56:14.133252 22489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.247656 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22427
I20260927 22:56:14.276980 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22501
I20260927 22:56:14.278167 22563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.390384 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22501
I20260927 22:56:14.408658 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22575
I20260927 22:56:14.409775 22637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:14.518128 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22575
2026-09-27T22:56:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4636 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:14 dist-test-slave-40vx krb5kdc[22720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:14 dist-test-slave-40vx krb5kdc[22720](info): set up 2 sockets
Sep 27 22:56:14 dist-test-slave-40vx krb5kdc[22720](info): commencing operation
krb5kdc: starting...
W20260927 22:56:15.616047 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:56:15 dist-test-slave-40vx krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:56:15.842947 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42617
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:42617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:16.161549 22733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:16.161818 22733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:16.161990 22733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:16.168984 22733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:16.169092 22733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:16.169203 22733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:16.169298 22733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:16.169379 22733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:16.179000 22733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:42617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:42617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:16.180374 22733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:16.181654 22733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:16.193387 22741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:16.193349 22738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:16.193421 22739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:16.194334 22733 server_base.cc:1034] running on GCE node
I20260927 22:56:16.195107 22733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:16.196501 22733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:16.197728 22733 hybrid_clock.cc:648] HybridClock initialized: now 1790549776197699 us; error 55 us; skew 500 ppm
I20260927 22:56:16.198083 22733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:16 dist-test-slave-40vx krb5kdc[22720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549776, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:16.201164 22733 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:16.202600 22733 webserver.cc:460] Webserver started at http://127.8.204.190:39715/ using document root <none> and password file <none>
I20260927 22:56:16.203336 22733 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:16.203473 22733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:16.203770 22733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:16.206667 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "b45b99125d264463a795242fcb6c0fc1"
format_stamp: "Formatted at 2026-09-27 22:56:16 on dist-test-slave-40vx"
server_key: "570ebb07d3061eaf3d71fe83c118bcbe"
server_key_iv: "26eb346fb9f47e0cf1bc572bcb3a3844"
server_key_version: "encryptionkey@0"
I20260927 22:56:16.207500 22733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "b45b99125d264463a795242fcb6c0fc1"
format_stamp: "Formatted at 2026-09-27 22:56:16 on dist-test-slave-40vx"
server_key: "570ebb07d3061eaf3d71fe83c118bcbe"
server_key_iv: "26eb346fb9f47e0cf1bc572bcb3a3844"
server_key_version: "encryptionkey@0"
I20260927 22:56:16.212550 22733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:56:16.215888 22748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:16.217227 22733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:16.217427 22733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "b45b99125d264463a795242fcb6c0fc1"
format_stamp: "Formatted at 2026-09-27 22:56:16 on dist-test-slave-40vx"
server_key: "570ebb07d3061eaf3d71fe83c118bcbe"
server_key_iv: "26eb346fb9f47e0cf1bc572bcb3a3844"
server_key_version: "encryptionkey@0"
I20260927 22:56:16.217630 22733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:16.232986 22733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:16.237293 22733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:16.237588 22733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:16.255975 22733 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:42617
I20260927 22:56:16.255995 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:42617 every 8 connection(s)
I20260927 22:56:16.257328 22733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:16.261063 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:16.263648 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22733
I20260927 22:56:16.263851 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:56:16.264521 9010 external_mini_cluster.cc:1428] Setting key 7d24912df92c3485175bd4a9eb329694
I20260927 22:56:16.273938 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1: Bootstrap starting.
I20260927 22:56:16.277477 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:16.278456 22800 log.cc:824] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:56:16 dist-test-slave-40vx krb5kdc[22720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:16.281780 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1: No bootstrap required, opened a new log
I20260927 22:56:16.288854 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:16.266605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.8.204.190:42617)
0927 22:56:16.267430 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:16.267445 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:16.267494 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:56:16.269783 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:16.269812 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:16.270353 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:16.270801 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:16.272242 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:16.274199 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:16.275638 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:16.276671 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:16.276865 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:16.281507 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:16.281514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:16.281528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:16.281569 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:16.284697 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:16.285577 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:16.285583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:16.285588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:16.285743 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:16.286155 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:16.286158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:16.286160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:16.286680 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:16.286940 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:16.287427 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 22:56:16.287828 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":369,"threads_started":1}
I20260927 22:56:16.290669 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b99125d264463a795242fcb6c0fc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42617 } }
I20260927 22:56:16.291110 22800 raft_consensus.cc:374] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:16.291239 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45b99125d264463a795242fcb6c0fc1, State: Initialized, Role: FOLLOWER
I20260927 22:56:16.291932 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b99125d264463a795242fcb6c0fc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42617 } }
I20260927 22:56:16.292246 22800 raft_consensus.cc:388] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:16.292407 22800 raft_consensus.cc:482] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:16.292586 22800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:16.292987 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:56:16.294836 22800 raft_consensus.cc:504] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b99125d264463a795242fcb6c0fc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42617 } }
I20260927 22:56:16.295434 22800 leader_election.cc:302] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45b99125d264463a795242fcb6c0fc1; no voters:
I20260927 22:56:16.296062 22800 leader_election.cc:288] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:16.296164 22805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:16.296957 22805 raft_consensus.cc:686] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [term 1 LEADER]: Becoming Leader. State: Replica: b45b99125d264463a795242fcb6c0fc1, State: Running, Role: LEADER
I20260927 22:56:16.297436 22805 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b99125d264463a795242fcb6c0fc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42617 } }
I20260927 22:56:16.298197 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:16.300369 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45b99125d264463a795242fcb6c0fc1. Latest consensus state: current_term: 1 leader_uuid: "b45b99125d264463a795242fcb6c0fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b99125d264463a795242fcb6c0fc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42617 } } }
I20260927 22:56:16.300897 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:16.300791 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45b99125d264463a795242fcb6c0fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45b99125d264463a795242fcb6c0fc1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 42617 } } }
I20260927 22:56:16.301793 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:16.306174 22813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:16.310254 22813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:16.318231 22813 catalog_manager.cc:1347] Generated new cluster ID: f4124bc56823426586329232ba4ff81a
I20260927 22:56:16.318336 22813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:16.329385 22813 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:16.331135 22813 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:16.345528 22813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b45b99125d264463a795242fcb6c0fc1: Generated new TSK 0
I20260927 22:56:16.346535 22813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:56:16.404920 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22733
I20260927 22:56:16.406131 22795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:16.503002 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22733
2026-09-27T22:56:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1977 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:16 dist-test-slave-40vx krb5kdc[22828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:16 dist-test-slave-40vx krb5kdc[22828](info): set up 2 sockets
Sep 27 22:56:16 dist-test-slave-40vx krb5kdc[22828](info): commencing operation
krb5kdc: starting...
W20260927 22:56:17.587848 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:56:17 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:56:17.818019 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40621
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:18.138550 22841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:18.138820 22841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:18.139014 22841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:18.146250 22841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:18.146353 22841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:18.146467 22841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:18.146569 22841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:18.146652 22841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:18.156486 22841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:18.157759 22841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:18.159034 22841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:18.170487 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:18.170708 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:18.170974 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:18.171310 22841 server_base.cc:1034] running on GCE node
I20260927 22:56:18.172091 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:18.173434 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:18.174671 22841 hybrid_clock.cc:648] HybridClock initialized: now 1790549778174622 us; error 56 us; skew 500 ppm
I20260927 22:56:18.175004 22841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:18 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:18.182909 22841 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:18.184423 22841 webserver.cc:460] Webserver started at http://127.8.204.190:39063/ using document root <none> and password file <none>
I20260927 22:56:18.185102 22841 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:18.185258 22841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:18.185565 22841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:18.188512 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "a1e86bb2d76d4e32861fe89bece75632"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "dfa21dc5e19edfc1a446ddab91ef0e67"
server_key_iv: "cff4f9dce15dcc85c673465d3eaad740"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.189260 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "a1e86bb2d76d4e32861fe89bece75632"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "dfa21dc5e19edfc1a446ddab91ef0e67"
server_key_iv: "cff4f9dce15dcc85c673465d3eaad740"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.194135 22841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:56:18.197382 22856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:18.198678 22841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:56:18.198853 22841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "a1e86bb2d76d4e32861fe89bece75632"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "dfa21dc5e19edfc1a446ddab91ef0e67"
server_key_iv: "cff4f9dce15dcc85c673465d3eaad740"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.199043 22841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:18.224359 22841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:18.228299 22841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:18.228588 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:18.246462 22841 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40621
I20260927 22:56:18.246508 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40621 every 8 connection(s)
I20260927 22:56:18.247818 22841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:18.249034 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22841
I20260927 22:56:18.249262 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:56:18.250135 9010 external_mini_cluster.cc:1428] Setting key f58837efcbb4f5eb8e6cf781bbc5244d
I20260927 22:56:18.252408 22908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:18.263042 22908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap starting.
Sep 27 22:56:18 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:18.266960 22908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:18.268329 22908 log.cc:824] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:18.271762 22908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: No bootstrap required, opened a new log
I20260927 22:56:18.273464 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:18.252183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.8.204.190:40621)
0927 22:56:18.252945 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:18.252956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:18.253021 (+ 65us) server_negotiation.cc:407] Connection header received
0927 22:56:18.254644 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:18.254663 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:18.255026 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:18.255431 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:18.256813 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.258140 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.259518 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.260186 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.260379 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:18.264700 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:18.264722 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:18.264740 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:18.264788 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:18.268687 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:18.269637 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:18.269643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:18.269648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:18.269885 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:18.270274 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:18.270279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:18.270282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:18.271006 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:18.271336 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:18.271751 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 22:56:18.272192 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":349,"threads_started":1}
I20260927 22:56:18.280187 22908 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:18.280609 22908 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:18.280727 22908 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e86bb2d76d4e32861fe89bece75632, State: Initialized, Role: FOLLOWER
I20260927 22:56:18.281322 22908 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:18.281554 22908 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:18.281704 22908 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:18.281878 22908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:18.284145 22908 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:18.284726 22908 leader_election.cc:302] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e86bb2d76d4e32861fe89bece75632; no voters:
I20260927 22:56:18.285275 22908 leader_election.cc:288] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:18.285403 22913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:18.286098 22913 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 LEADER]: Becoming Leader. State: Replica: a1e86bb2d76d4e32861fe89bece75632, State: Running, Role: LEADER
I20260927 22:56:18.286581 22913 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:18.287389 22908 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:18.289930 22915 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e86bb2d76d4e32861fe89bece75632. Latest consensus state: current_term: 1 leader_uuid: "a1e86bb2d76d4e32861fe89bece75632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } } }
I20260927 22:56:18.290264 22915 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:18.291527 22921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:18.291438 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1e86bb2d76d4e32861fe89bece75632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } } }
I20260927 22:56:18.291775 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:18.296259 22921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:18.304388 22921 catalog_manager.cc:1347] Generated new cluster ID: e4c396da7a49417b814e021ec65b8ec0
I20260927 22:56:18.304502 22921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:18.320369 22921 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:18.321647 22921 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:18.329138 22921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Generated new TSK 0
I20260927 22:56:18.330133 22921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:56:18.391034 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40621
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:18.712954 22934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:18.713225 22934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:18.713346 22934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:18.720623 22934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:18.720731 22934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:18.720902 22934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:56:18.730892 22934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:18.732223 22934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:18.733596 22934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:18.747229 22939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:18.747262 22942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:18.747627 22940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:18.748145 22934 server_base.cc:1034] running on GCE node
I20260927 22:56:18.748715 22934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:18.749598 22934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:18.750804 22934 hybrid_clock.cc:648] HybridClock initialized: now 1790549778750750 us; error 53 us; skew 500 ppm
I20260927 22:56:18.751129 22934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:18 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:18.754384 22934 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:56:18.755831 22934 webserver.cc:460] Webserver started at http://127.8.204.129:35017/ using document root <none> and password file <none>
I20260927 22:56:18.756511 22934 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:18.756660 22934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:18.756959 22934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:18.759824 22934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "31e574436ac3f186b2f1bda9baf20a6f"
server_key_iv: "3c65d292539c713dea8b64b73413467c"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.760565 22934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "31e574436ac3f186b2f1bda9baf20a6f"
server_key_iv: "3c65d292539c713dea8b64b73413467c"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.765506 22934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:56:18.768975 22949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:18.770335 22934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:18.770503 22934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "31e574436ac3f186b2f1bda9baf20a6f"
server_key_iv: "3c65d292539c713dea8b64b73413467c"
server_key_version: "encryptionkey@0"
I20260927 22:56:18.770696 22934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:18.789287 22934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:18.794034 22934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:18.794332 22934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:18.795245 22934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:18.796578 22934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:18.796679 22934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:18.796828 22934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:18.796900 22934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:18.830384 22934 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41957
I20260927 22:56:18.830430 23061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41957 every 8 connection(s)
I20260927 22:56:18.831727 22934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:18.832695 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 22934
I20260927 22:56:18.832908 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:18.833753 9010 external_mini_cluster.cc:1428] Setting key 1bcf5e6940e9dbac98db978390d82045
Sep 27 22:56:18 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:18.853637 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:18.834647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:50791 (local address 127.8.204.190:40621)
0927 22:56:18.835101 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:18.835105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:18.836354 (+ 1249us) server_negotiation.cc:407] Connection header received
0927 22:56:18.838140 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:18.838149 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:18.838248 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:18.838371 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:18.839869 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.841031 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.842558 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.842970 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.843071 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:18.847737 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:18.847742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:18.847746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:18.847778 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:18.850250 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:18.851170 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:18.851174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:18.851176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:18.851345 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:18.851816 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:18.851819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:18.851821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:18.852074 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:18.852192 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:18.853066 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 22:56:18.853341 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":249,"threads_started":1}
I20260927 22:56:18.855044 23065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:18.835263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:50791)
0927 22:56:18.836178 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:18.836229 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:18.837878 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:18.838562 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:18.838570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:18.839069 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:18.839696 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:18.839713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.841183 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.841187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:18.842400 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:18.842411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:18.843217 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:18.843225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:18.843603 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:18.844317 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:18.844355 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:18.847516 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:18.850389 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:18.850402 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:18.850423 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:18.851039 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:18.851448 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:18.851455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:18.851462 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:18.851706 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:18.852204 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:18.852214 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:18.852670 (+ 456us) client_negotiation.cc:769] Sending connection context
0927 22:56:18.852978 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 22:56:18.853561 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":393,"threads_started":1}
I20260927 22:56:18.856709 23062 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40621
I20260927 22:56:18.857048 23062 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:18.857963 23062 heartbeater.cc:511] Master 127.8.204.190:40621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:56:18.860185 22873 ts_manager.cc:194] Registered new tserver with Master: e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:56:18.861889 22873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:50791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:56:18.886737 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40621
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:19.208026 23068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:19.208299 23068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:19.208474 23068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:19.215523 23068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:19.215654 23068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:19.215849 23068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:56:19.225212 23068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:19.226490 23068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:19.227844 23068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:19.240849 23074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:19.240805 23076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:19.240870 23073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:19.241633 23068 server_base.cc:1034] running on GCE node
I20260927 22:56:19.242220 23068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:19.243135 23068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:19.244359 23068 hybrid_clock.cc:648] HybridClock initialized: now 1790549779244312 us; error 65 us; skew 500 ppm
I20260927 22:56:19.244658 23068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:19 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549779, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:19.248232 23068 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:56:19.249871 23068 webserver.cc:460] Webserver started at http://127.8.204.130:34193/ using document root <none> and password file <none>
I20260927 22:56:19.250694 23068 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:19.250813 23068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:19.251108 23068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:19.254984 23068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "b5eeaf9486c04625b57af395ec03d498"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4b439a4e004846091849866532a65060"
server_key_iv: "7bfbb54373a96dfde42805d0fd8ab35d"
server_key_version: "encryptionkey@0"
I20260927 22:56:19.255960 23068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "b5eeaf9486c04625b57af395ec03d498"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4b439a4e004846091849866532a65060"
server_key_iv: "7bfbb54373a96dfde42805d0fd8ab35d"
server_key_version: "encryptionkey@0"
I20260927 22:56:19.262209 23068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:56:19.266295 23083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:19.267838 23068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:19.268023 23068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "b5eeaf9486c04625b57af395ec03d498"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4b439a4e004846091849866532a65060"
server_key_iv: "7bfbb54373a96dfde42805d0fd8ab35d"
server_key_version: "encryptionkey@0"
I20260927 22:56:19.268239 23068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:19.281339 23068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:19.285336 23068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:19.285627 23068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:19.286546 23068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:19.288043 23068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:19.288115 23068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:19.288266 23068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:19.288319 23068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:19.324937 23068 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:45883
I20260927 22:56:19.324975 23195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:45883 every 8 connection(s)
I20260927 22:56:19.326229 23068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:19.328557 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23068
I20260927 22:56:19.328786 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:19.329362 9010 external_mini_cluster.cc:1428] Setting key 6169b0642a626c233263ac4f188c7a4a
Sep 27 22:56:19 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549779, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:19.346953 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.328930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:55295 (local address 127.8.204.190:40621)
0927 22:56:19.329146 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:19.329152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:19.330500 (+ 1348us) server_negotiation.cc:407] Connection header received
0927 22:56:19.332571 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:19.332576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:19.332668 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:19.332786 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:19.334263 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.334967 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.336606 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.337028 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.337181 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.341168 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:19.341173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:19.341176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:19.341209 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:19.343923 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:19.344666 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:19.344673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:19.344681 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:19.344803 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:19.345232 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:19.345237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:19.345240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:19.345504 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:19.345660 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:19.346354 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 22:56:19.346582 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:56:19.348066 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.329516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:55295)
0927 22:56:19.330316 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:19.330369 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:19.332302 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:19.333020 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:19.333028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:19.333529 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:19.334095 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.334111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.335133 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.335137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.336449 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.336462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.337261 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.337268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.337504 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.338191 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:19.338223 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:19.340919 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:19.344087 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:19.344096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:19.344113 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:19.344534 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:19.344915 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:19.344920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:19.344924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:19.345115 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:19.345633 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:19.345648 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:19.345997 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 22:56:19.346256 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 22:56:19.346815 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":338,"threads_started":1}
I20260927 22:56:19.349582 23196 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40621
I20260927 22:56:19.349951 23196 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:19.350919 23196 heartbeater.cc:511] Master 127.8.204.190:40621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:56:19.352655 22873 ts_manager.cc:194] Registered new tserver with Master: b5eeaf9486c04625b57af395ec03d498 (127.8.204.130:45883)
I20260927 22:56:19.353499 22873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:55295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:56:19.382326 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40621
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:19.701345 23201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:19.701632 23201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:19.701817 23201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:19.709196 23201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:19.709301 23201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:19.709411 23201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:56:19.718976 23201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:19.720234 23201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:19.721460 23201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:19.733978 23206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:19.734067 23207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:19.734088 23209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:19.735577 23201 server_base.cc:1034] running on GCE node
I20260927 22:56:19.736214 23201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:19.737121 23201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:19.738341 23201 hybrid_clock.cc:648] HybridClock initialized: now 1790549779738299 us; error 39 us; skew 500 ppm
I20260927 22:56:19.738660 23201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:19 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549779, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:19.741950 23201 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:56:19.743624 23201 webserver.cc:460] Webserver started at http://127.8.204.131:41887/ using document root <none> and password file <none>
I20260927 22:56:19.744334 23201 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:19.744479 23201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:19.744786 23201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:19.747802 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "61f5d07333094f3c84c20305302580b4"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4968dcd23542053ef315675d44a97e77"
server_key_iv: "f2c9b6e68fe997d36b494cd55f81a41b"
server_key_version: "encryptionkey@0"
I20260927 22:56:19.748574 23201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "61f5d07333094f3c84c20305302580b4"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4968dcd23542053ef315675d44a97e77"
server_key_iv: "f2c9b6e68fe997d36b494cd55f81a41b"
server_key_version: "encryptionkey@0"
I20260927 22:56:19.753662 23201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:56:19.756996 23216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:19.758348 23201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:19.758530 23201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "61f5d07333094f3c84c20305302580b4"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4968dcd23542053ef315675d44a97e77"
server_key_iv: "f2c9b6e68fe997d36b494cd55f81a41b"
server_key_version: "encryptionkey@0"
I20260927 22:56:19.758724 23201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:19.790745 23201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:19.795189 23201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:19.795521 23201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:19.796530 23201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:19.797955 23201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:19.798062 23201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:19.798228 23201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:19.798300 23201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:19.835065 23201 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:34967
I20260927 22:56:19.835098 23328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:34967 every 8 connection(s)
I20260927 22:56:19.836460 23201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:19.844538 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23201
I20260927 22:56:19.844719 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:19.845178 9010 external_mini_cluster.cc:1428] Setting key 6342f6f81f682f14d93f4d776e83545d
Sep 27 22:56:19 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549779, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:19.855959 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.838838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:39739 (local address 127.8.204.190:40621)
0927 22:56:19.839039 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:19.839045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:19.840277 (+ 1232us) server_negotiation.cc:407] Connection header received
0927 22:56:19.842096 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:19.842101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:19.842195 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:19.842287 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:19.843831 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.844627 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.846412 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.847057 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.847210 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.850847 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:19.850852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:19.850855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:19.850890 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:19.853277 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:19.853929 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:19.853933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:19.853935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:19.854047 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:19.854413 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:19.854416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:19.854418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:19.854679 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:19.854785 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:19.855512 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 22:56:19.855713 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:56:19.856963 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.839449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:39739)
0927 22:56:19.840107 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:19.840151 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:19.841825 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:19.842502 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:19.842510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:19.843007 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:19.843666 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.843687 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.844844 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.844848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.846287 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.846299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.847330 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.847340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.847646 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.848426 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:19.848467 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:19.850654 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:19.853400 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:19.853407 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:19.853420 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:19.853815 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:19.854137 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:19.854139 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:19.854142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:19.854318 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:19.854779 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:19.854784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:19.855082 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 22:56:19.855388 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 22:56:19.855857 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":260,"threads_started":1}
I20260927 22:56:19.858363 23329 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40621
I20260927 22:56:19.858619 23329 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:19.859495 23329 heartbeater.cc:511] Master 127.8.204.190:40621 requested a full tablet report, sending...
I20260927 22:56:19.860978 22873 ts_manager.cc:194] Registered new tserver with Master: 61f5d07333094f3c84c20305302580b4 (127.8.204.131:34967)
I20260927 22:56:19.861805 22873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:39739
I20260927 22:56:19.864765 23062 heartbeater.cc:503] Master 127.8.204.190:40621 was elected leader, sending a full tablet report...
I20260927 22:56:19.872937 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:19.889233 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.877297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.8.204.190:40621)
0927 22:56:19.877535 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:19.877541 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:19.877805 (+ 264us) server_negotiation.cc:407] Connection header received
0927 22:56:19.877934 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:19.877937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:19.877998 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:19.878075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:19.879434 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.880187 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.881414 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.881789 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.881879 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.884176 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:19.884181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:19.884184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:19.884216 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:19.886564 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:19.887256 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:19.887261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:19.887263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:19.887447 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:19.887849 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:19.887852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:19.887854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:19.888259 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:19.888449 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:19.888741 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 22:56:19.888905 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:56:19.893312 22873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:19.900098 22873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:19.926839 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.912695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.8.204.129:41957)
0927 22:56:19.914574 (+ 1879us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:19.914582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:19.914940 (+ 358us) server_negotiation.cc:407] Connection header received
0927 22:56:19.915298 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:19.915311 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:19.915613 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:19.915831 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:19.917329 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.919566 (+ 2237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.924127 (+ 4561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.925399 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.925535 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.925759 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:19.926327 (+ 568us) server_negotiation.cc:299] Negotiation successful
0927 22:56:19.926500 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1686,"thread_start_us":1540,"threads_started":1}
I20260927 22:56:19.927119 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.912082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:34967 (local address 127.0.0.1:56786)
0927 22:56:19.913007 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:19.913062 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:19.913286 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:19.915888 (+ 2602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:19.915899 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:19.915940 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:19.916551 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.916568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.921925 (+ 5357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.921935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.924458 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.924471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.926163 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.926177 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.926365 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.926405 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:56:19.926511 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:56:19.926695 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":25,"thread_start_us":845,"threads_started":1}
I20260927 22:56:19.927455 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.912412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.8.204.131:34967)
0927 22:56:19.915091 (+ 2679us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:19.915099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:19.915130 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:56:19.915248 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:19.915259 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:19.915600 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:19.917840 (+ 2240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:19.918173 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.921737 (+ 3564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.924627 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.925960 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.926106 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.926339 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:19.926551 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:56:19.926712 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2475,"thread_start_us":2422,"threads_started":1}
I20260927 22:56:19.927873 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.912272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55328)
0927 22:56:19.914821 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:19.914843 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:19.915046 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:19.916218 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:19.916226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:19.916265 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:19.917086 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.917099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.919818 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.919826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.923164 (+ 3338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.923184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.925829 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.925835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.926051 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.926112 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 22:56:19.927155 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0927 22:56:19.927325 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2426,"mutex_wait_us":4,"thread_start_us":2289,"threads_started":1}
I20260927 22:56:19.931731 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.914389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:45883 (local address 127.0.0.1:46794)
0927 22:56:19.923865 (+ 9476us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:19.923882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:19.923988 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:19.924647 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:19.924651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:19.924677 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:19.925432 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.925454 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.927357 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.927362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.929281 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:19.929293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.930955 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.930961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:19.931126 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.931156 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:56:19.931254 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 22:56:19.931396 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9297,"thread_start_us":9108,"threads_started":1}
I20260927 22:56:19.931803 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:19.914641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46794 (local address 127.8.204.130:45883)
0927 22:56:19.924091 (+ 9450us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:19.924100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:19.924126 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:56:19.924211 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:19.924218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:19.924460 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:19.924646 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:19.925634 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.927198 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.929427 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:19.930749 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:19.930897 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:19.931087 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:19.931329 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 22:56:19.931497 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9280,"thread_start_us":9177,"threads_started":1}
I20260927 22:56:19.931659 22997 tablet_service.cc:1467] Processing CreateTablet for tablet c4c00043977a47a59b0c4687ef88570e (DEFAULT_TABLE table=test-table [id=72e78cad3de34738865e52de5924d035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:19.931660 23264 tablet_service.cc:1467] Processing CreateTablet for tablet c4c00043977a47a59b0c4687ef88570e (DEFAULT_TABLE table=test-table [id=72e78cad3de34738865e52de5924d035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:19.933159 22997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4c00043977a47a59b0c4687ef88570e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:19.933157 23264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4c00043977a47a59b0c4687ef88570e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:19.934780 23131 tablet_service.cc:1467] Processing CreateTablet for tablet c4c00043977a47a59b0c4687ef88570e (DEFAULT_TABLE table=test-table [id=72e78cad3de34738865e52de5924d035]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:19.936048 23131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4c00043977a47a59b0c4687ef88570e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:19.948029 23348 tablet_bootstrap.cc:492] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Bootstrap starting.
I20260927 22:56:19.950136 23349 tablet_bootstrap.cc:492] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Bootstrap starting.
I20260927 22:56:19.951607 23347 tablet_bootstrap.cc:492] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Bootstrap starting.
I20260927 22:56:19.953107 23348 tablet_bootstrap.cc:654] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:19.953905 23349 tablet_bootstrap.cc:654] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:19.954183 23348 log.cc:824] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:19.955066 23347 tablet_bootstrap.cc:654] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:19.955107 23349 log.cc:824] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:19.956451 23347 log.cc:824] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:19.957232 23348 tablet_bootstrap.cc:492] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: No bootstrap required, opened a new log
I20260927 22:56:19.957511 23348 ts_tablet_manager.cc:1397] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260927 22:56:19.958102 23349 tablet_bootstrap.cc:492] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: No bootstrap required, opened a new log
I20260927 22:56:19.958417 23349 ts_tablet_manager.cc:1397] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260927 22:56:19.959333 23347 tablet_bootstrap.cc:492] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: No bootstrap required, opened a new log
I20260927 22:56:19.959589 23347 ts_tablet_manager.cc:1397] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 22:56:19.965286 23348 raft_consensus.cc:348] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:19.965677 23348 raft_consensus.cc:374] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:19.965788 23348 raft_consensus.cc:729] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f5d07333094f3c84c20305302580b4, State: Initialized, Role: FOLLOWER
I20260927 22:56:19.966404 23348 consensus_queue.cc:260] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:19.967546 23329 heartbeater.cc:503] Master 127.8.204.190:40621 was elected leader, sending a full tablet report...
I20260927 22:56:19.968225 23348 ts_tablet_manager.cc:1428] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 22:56:19.969393 23349 raft_consensus.cc:348] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:19.969805 23349 raft_consensus.cc:374] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:19.969909 23349 raft_consensus.cc:729] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7aa8c7767df44538fbef2a33a1fd1b4, State: Initialized, Role: FOLLOWER
I20260927 22:56:19.970232 23347 raft_consensus.cc:348] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:19.970654 23347 raft_consensus.cc:374] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:19.970507 23349 consensus_queue.cc:260] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:19.970760 23347 raft_consensus.cc:729] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5eeaf9486c04625b57af395ec03d498, State: Initialized, Role: FOLLOWER
I20260927 22:56:19.971452 23347 consensus_queue.cc:260] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:19.972910 23349 ts_tablet_manager.cc:1428] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260927 22:56:19.975435 23196 heartbeater.cc:503] Master 127.8.204.190:40621 was elected leader, sending a full tablet report...
I20260927 22:56:19.975998 23347 ts_tablet_manager.cc:1428] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260927 22:56:20.078176 23197 tablet.cc:2369] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:20.084056 23063 tablet.cc:2369] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:20.088024 23330 tablet.cc:2369] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:20.309176 23353 raft_consensus.cc:482] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:20.309522 23353 raft_consensus.cc:504] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:20.311348 23353 leader_election.cc:288] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957), b5eeaf9486c04625b57af395ec03d498 (127.8.204.130:45883)
I20260927 22:56:20.317740 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.311964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38597 (local address 127.8.204.130:45883)
0927 22:56:20.312176 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.312183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.312208 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:56:20.312269 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.312273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.312330 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.312446 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:20.312944 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.313958 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.315841 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.316893 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.317022 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.317079 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.317428 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.317518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:20.317838 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.311731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:45883 (local address 127.8.204.131:38597)
0927 22:56:20.311903 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:20.311924 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:20.312057 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:20.312477 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:20.312482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:20.312509 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:20.312778 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:20.312793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.314108 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.314114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:20.315668 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:20.315681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.317051 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.317056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:20.317255 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.317293 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:56:20.317390 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:56:20.317537 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 22:56:20.318559 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.311745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.8.204.131:35497)
0927 22:56:20.312570 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:20.312583 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:20.312691 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:20.313064 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:20.313068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:20.313093 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:20.313460 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:20.313473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.314741 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.314745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:20.316383 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:20.316398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.317857 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.317866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:20.318041 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.318077 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:56:20.318165 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:56:20.318292 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":145,"thread_start_us":345,"threads_started":1}
I20260927 22:56:20.318957 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.311763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:35497 (local address 127.8.204.129:41957)
0927 22:56:20.311969 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.311975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.312677 (+ 702us) server_negotiation.cc:407] Connection header received
0927 22:56:20.312832 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.312839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.312934 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.313007 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:20.313625 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.314584 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.316563 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.317640 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.317788 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.317867 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.318590 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.318733 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:56:20.319013 23151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c00043977a47a59b0c4687ef88570e" candidate_uuid: "61f5d07333094f3c84c20305302580b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5eeaf9486c04625b57af395ec03d498" is_pre_election: true
I20260927 22:56:20.319619 23151 raft_consensus.cc:2454] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f5d07333094f3c84c20305302580b4 in term 0.
I20260927 22:56:20.320010 23017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c00043977a47a59b0c4687ef88570e" candidate_uuid: "61f5d07333094f3c84c20305302580b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" is_pre_election: true
I20260927 22:56:20.320324 23220 leader_election.cc:302] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f5d07333094f3c84c20305302580b4, b5eeaf9486c04625b57af395ec03d498; no voters:
I20260927 22:56:20.320581 23017 raft_consensus.cc:2454] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 61f5d07333094f3c84c20305302580b4 in term 0.
I20260927 22:56:20.320749 23353 raft_consensus.cc:2781] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:20.320904 23353 raft_consensus.cc:482] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:20.321020 23353 raft_consensus.cc:3037] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:20.323228 23353 raft_consensus.cc:504] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:20.323899 23353 leader_election.cc:288] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [CANDIDATE]: Term 1 election: Requested vote from peers e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957), b5eeaf9486c04625b57af395ec03d498 (127.8.204.130:45883)
I20260927 22:56:20.324340 23017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c00043977a47a59b0c4687ef88570e" candidate_uuid: "61f5d07333094f3c84c20305302580b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
I20260927 22:56:20.324455 23151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4c00043977a47a59b0c4687ef88570e" candidate_uuid: "61f5d07333094f3c84c20305302580b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5eeaf9486c04625b57af395ec03d498"
I20260927 22:56:20.324543 23017 raft_consensus.cc:3037] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:20.324695 23151 raft_consensus.cc:3037] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:20.327832 23017 raft_consensus.cc:2454] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f5d07333094f3c84c20305302580b4 in term 1.
I20260927 22:56:20.328040 23151 raft_consensus.cc:2454] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 61f5d07333094f3c84c20305302580b4 in term 1.
I20260927 22:56:20.328353 23218 leader_election.cc:302] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f5d07333094f3c84c20305302580b4, e7aa8c7767df44538fbef2a33a1fd1b4; no voters:
I20260927 22:56:20.328677 23353 raft_consensus.cc:2781] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:20.329183 23353 raft_consensus.cc:686] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 1 LEADER]: Becoming Leader. State: Replica: 61f5d07333094f3c84c20305302580b4, State: Running, Role: LEADER
I20260927 22:56:20.329769 23353 consensus_queue.cc:237] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:20.334439 22872 catalog_manager.cc:5495] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 61f5d07333094f3c84c20305302580b4 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "61f5d07333094f3c84c20305302580b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } health_report { overall_health: UNKNOWN } } }
I20260927 22:56:20.378975 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.372892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.8.204.131:34967)
0927 22:56:20.373147 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.373153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.373181 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:56:20.373281 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.373284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.373368 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.373590 (+ 222us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:20.374024 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.375228 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.376865 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.377263 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.377382 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.377874 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:20.378402 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:20.378540 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.378645 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.378713 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:56:20.387091 23151 raft_consensus.cc:1261] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Refusing update from remote peer 61f5d07333094f3c84c20305302580b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:20.387128 23017 raft_consensus.cc:1261] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Refusing update from remote peer 61f5d07333094f3c84c20305302580b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:20.388202 23357 consensus_queue.cc:1035] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:20.388648 23353 consensus_queue.cc:1035] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:20.401019 23362 mvcc.cc:205] Tried to move back new op lower bound from 7334091900455645184 to 7334091900234899456. Current Snapshot: MvccSnapshot[applied={T|T < 7334091900455645184}]
I20260927 22:56:20.412084 22873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:20.412395 22873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:20.415822 22873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Sending DeleteTablet for 3 replicas of tablet c4c00043977a47a59b0c4687ef88570e
I20260927 22:56:20.416998 22997 tablet_service.cc:1514] Processing DeleteTablet for tablet c4c00043977a47a59b0c4687ef88570e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:20 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:55328
I20260927 22:56:20.417093 23131 tablet_service.cc:1514] Processing DeleteTablet for tablet c4c00043977a47a59b0c4687ef88570e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:20 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:46794
I20260927 22:56:20.417802 23264 tablet_service.cc:1514] Processing DeleteTablet for tablet c4c00043977a47a59b0c4687ef88570e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:20 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:56786
I20260927 22:56:20.418020 23371 tablet_replica.cc:330] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: stopping tablet replica
I20260927 22:56:20.418440 23371 raft_consensus.cc:2229] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:20.418932 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22934
I20260927 22:56:20.418953 23371 raft_consensus.cc:2258] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:20.419706 23370 tablet_replica.cc:330] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: stopping tablet replica
I20260927 22:56:20.420078 23370 raft_consensus.cc:2229] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:20.420535 23370 raft_consensus.cc:2258] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:20.422417 23371 ts_tablet_manager.cc:1905] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:20.429698 23371 ts_tablet_manager.cc:1918] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:20.430353 23372 tablet_replica.cc:330] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: stopping tablet replica
I20260927 22:56:20.430967 23372 raft_consensus.cc:2229] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:20.431634 23370 ts_tablet_manager.cc:1905] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:20.431804 23372 raft_consensus.cc:2258] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:56:20 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:56:20.434118 23372 ts_tablet_manager.cc:1905] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:20.439085 23370 ts_tablet_manager.cc:1918] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:20.439368 23371 log.cc:1196] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/c4c00043977a47a59b0c4687ef88570e
I20260927 22:56:20.439975 23371 ts_tablet_manager.cc:1939] T c4c00043977a47a59b0c4687ef88570e P b5eeaf9486c04625b57af395ec03d498: Deleting consensus metadata
I20260927 22:56:20.441886 23372 ts_tablet_manager.cc:1918] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:20.441954 22860 catalog_manager.cc:4841] TS b5eeaf9486c04625b57af395ec03d498 (127.8.204.130:45883): tablet c4c00043977a47a59b0c4687ef88570e (table test-table [id=72e78cad3de34738865e52de5924d035]) successfully deleted
I20260927 22:56:20.442174 23372 log.cc:1196] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/c4c00043977a47a59b0c4687ef88570e
I20260927 22:56:20.442652 23344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.420088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.8.204.129:41957)
0927 22:56:20.420268 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.420274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.420397 (+ 123us) server_negotiation.cc:407] Connection header received
0927 22:56:20.420693 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.420702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.420807 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.420895 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:20.423240 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.424733 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.427479 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.428339 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.428468 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.434459 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:20.434472 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:20.434489 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:20.434536 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:20.438586 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:20.439915 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:20.439924 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:20.439927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:20.440100 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:20.440580 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:20.440586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:20.440589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:20.441023 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:20.441300 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.441930 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.442169 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:56:20.442392 23370 log.cc:1196] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/c4c00043977a47a59b0c4687ef88570e
I20260927 22:56:20.443325 23370 ts_tablet_manager.cc:1939] T c4c00043977a47a59b0c4687ef88570e P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting consensus metadata
I20260927 22:56:20.443615 23372 ts_tablet_manager.cc:1939] T c4c00043977a47a59b0c4687ef88570e P 61f5d07333094f3c84c20305302580b4: Deleting consensus metadata
I20260927 22:56:20.443984 23057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:20.445556 22860 catalog_manager.cc:4841] TS 61f5d07333094f3c84c20305302580b4 (127.8.204.131:34967): tablet c4c00043977a47a59b0c4687ef88570e (table test-table [id=72e78cad3de34738865e52de5924d035]) successfully deleted
I20260927 22:56:20.571003 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22934
I20260927 22:56:20.571648 22858 catalog_manager.cc:4841] TS e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957): tablet c4c00043977a47a59b0c4687ef88570e (table test-table [id=72e78cad3de34738865e52de5924d035]) successfully deleted
I20260927 22:56:20.590543 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23068
Sep 27 22:56:20 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:56:20.606513 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.591364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.8.204.130:45883)
0927 22:56:20.591539 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.591547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.591570 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:56:20.591824 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.591828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.591896 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.592002 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:20.593339 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.594137 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.595562 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.595960 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.596096 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.600567 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:20.600581 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:20.600597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:20.600632 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:20.603629 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:20.604419 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:20.604426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:20.604428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:20.604554 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:20.604943 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:20.604945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:20.604947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:20.605304 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:20.605613 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.605940 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.606159 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:56:20.607203 23191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:20.696623 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23068
I20260927 22:56:20.713959 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23201
Sep 27 22:56:20 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:56:20.730036 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:20.714778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.8.204.131:34967)
0927 22:56:20.714974 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:20.714982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:20.715008 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:56:20.715244 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:20.715248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:20.715369 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:20.715473 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:20.716794 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.717580 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.718880 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:20.719408 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:20.719550 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:20.724092 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:20.724103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:20.724122 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:20.724158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:20.727155 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:20.727974 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:20.727980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:20.727983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:20.728143 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:20.728578 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:20.728581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:20.728583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:20.728893 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:20.729151 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:20.729503 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 22:56:20.729659 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:56:20.730786 23324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:20.853408 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23201
I20260927 22:56:20.872782 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 22841
I20260927 22:56:20.874079 22903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:20.987018 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 22841
I20260927 22:56:21.008210 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40621
--webserver_interface=127.8.204.190
--webserver_port=39063
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:21.327042 23378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:21.327420 23378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:21.327594 23378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:21.334698 23378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:21.334861 23378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:21.334982 23378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:21.335086 23378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:21.335170 23378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:21.345142 23378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=39063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:21.346510 23378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:21.347891 23378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:21.370600 23387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:21.370589 23385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:21.370591 23384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:21.371467 23378 server_base.cc:1034] running on GCE node
I20260927 22:56:21.372318 23378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:21.373921 23378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:21.375168 23378 hybrid_clock.cc:648] HybridClock initialized: now 1790549781375121 us; error 54 us; skew 500 ppm
I20260927 22:56:21.375612 23378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:21 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549781, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:21.380072 23378 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:21.381821 23378 webserver.cc:460] Webserver started at http://127.8.204.190:39063/ using document root <none> and password file <none>
I20260927 22:56:21.382635 23378 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:21.382807 23378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:21.389195 23378 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:56:21.392567 23394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:21.394109 23378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:56:21.394253 23378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "a1e86bb2d76d4e32861fe89bece75632"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "dfa21dc5e19edfc1a446ddab91ef0e67"
server_key_iv: "cff4f9dce15dcc85c673465d3eaad740"
server_key_version: "encryptionkey@0"
I20260927 22:56:21.395378 23378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:21.426064 23378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:21.430615 23378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:21.430836 23378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:21.449458 23378 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40621
I20260927 22:56:21.449483 23445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40621 every 8 connection(s)
I20260927 22:56:21.450946 23378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:21.457727 23446 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:56:21.459722 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23378
I20260927 22:56:21.460515 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:41957
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:40621
--webserver_port=35017
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:21.461113 23446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap starting.
I20260927 22:56:21.480772 23446 log.cc:824] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:21.492628 23446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:56:21.493136 23446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap complete.
I20260927 22:56:21.503108 23446 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:21.503986 23446 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e86bb2d76d4e32861fe89bece75632, State: Initialized, Role: FOLLOWER
I20260927 22:56:21.504657 23446 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:21.504907 23446 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:21.505043 23446 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:21.505256 23446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:56:21.509115 23446 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:21.509625 23446 leader_election.cc:302] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e86bb2d76d4e32861fe89bece75632; no voters:
I20260927 22:56:21.510304 23446 leader_election.cc:288] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:56:21.510440 23450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:56:21.511693 23450 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 LEADER]: Becoming Leader. State: Replica: a1e86bb2d76d4e32861fe89bece75632, State: Running, Role: LEADER
I20260927 22:56:21.512308 23450 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:21.512709 23446 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:21.514714 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e86bb2d76d4e32861fe89bece75632. Latest consensus state: current_term: 2 leader_uuid: "a1e86bb2d76d4e32861fe89bece75632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } } }
I20260927 22:56:21.515085 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:21.515535 23451 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a1e86bb2d76d4e32861fe89bece75632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } } }
I20260927 22:56:21.515810 23451 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:21.520804 23456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:21.533524 23456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:21.536949 23456 catalog_manager.cc:1259] Loaded cluster ID: e4c396da7a49417b814e021ec65b8ec0
I20260927 22:56:21.537113 23456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:21.548362 23456 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:56:21.576103 23456 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:56:21.585438 23456 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:21.586864 23456 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Loaded TSK: 0
I20260927 22:56:21.588076 23456 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:56:21.838987 23448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:21.839429 23448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:21.839627 23448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:21.847431 23448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:21.847538 23448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:21.847731 23448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:56:21.857661 23448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:41957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=35017
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:21.859025 23448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:21.860526 23448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:21.875319 23474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:21.875504 23476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:21.876135 23473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:21.876849 23448 server_base.cc:1034] running on GCE node
I20260927 22:56:21.877542 23448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:21.878690 23448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:21.879928 23448 hybrid_clock.cc:648] HybridClock initialized: now 1790549781879861 us; error 54 us; skew 500 ppm
I20260927 22:56:21.880317 23448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:21 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549781, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:21.884366 23448 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:56:21.886088 23448 webserver.cc:460] Webserver started at http://127.8.204.129:35017/ using document root <none> and password file <none>
I20260927 22:56:21.887038 23448 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:21.887197 23448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:21.894222 23448 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:56:21.897425 23483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:21.899030 23448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:56:21.899209 23448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "31e574436ac3f186b2f1bda9baf20a6f"
server_key_iv: "3c65d292539c713dea8b64b73413467c"
server_key_version: "encryptionkey@0"
I20260927 22:56:21.900578 23448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:21.928730 23448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:21.934473 23448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:21.934769 23448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:21.936112 23448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:21.937897 23448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:21.938031 23448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:21.938145 23448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:21.938216 23448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:21.978277 23448 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41957
I20260927 22:56:21.978332 23595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41957 every 8 connection(s)
I20260927 22:56:21.979897 23448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:21.984287 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23448
I20260927 22:56:21.984917 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:45883
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:40621
--webserver_port=34193
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:56:21 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549781, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:22.007332 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:21.982940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:60555 (local address 127.8.204.190:40621)
0927 22:56:21.983513 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:21.983526 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:21.984298 (+ 772us) server_negotiation.cc:407] Connection header received
0927 22:56:21.986554 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:21.986575 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:21.987055 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:21.987487 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:21.988930 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.990942 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.993128 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.993754 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.993884 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:21.998560 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:21.998576 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:21.998597 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:21.998660 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:22.002305 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.003382 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.003389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.003396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.003557 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.004094 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.004097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.004099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.004548 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:22.004703 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.005804 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.006160 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":258,"threads_started":1}
I20260927 22:56:22.007777 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:21.983224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:60555)
0927 22:56:21.984084 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:21.984149 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:21.985991 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:21.987599 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:21.987607 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:21.988100 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:21.988692 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:21.988709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.991104 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.991107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:21.992950 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:21.992967 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:21.994000 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:21.994010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:21.994323 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:21.995049 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:21.995089 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:21.998185 (+ 3096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:22.002460 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:22.002473 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:22.002492 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:22.003195 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:22.003679 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:22.003695 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:22.003700 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:22.003971 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:22.004699 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:22.004709 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:22.005205 (+ 496us) client_negotiation.cc:769] Sending connection context
0927 22:56:22.005558 (+ 353us) client_negotiation.cc:240] Negotiation successful
0927 22:56:22.006247 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":339,"threads_started":1}
I20260927 22:56:22.010948 23596 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40621
I20260927 22:56:22.011245 23596 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:22.011981 23596 heartbeater.cc:511] Master 127.8.204.190:40621 requested a full tablet report, sending...
I20260927 22:56:22.014355 23411 ts_manager.cc:194] Registered new tserver with Master: e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957)
I20260927 22:56:22.016489 23411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:60555
W20260927 22:56:22.315898 23600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:22.316179 23600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:22.316287 23600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:22.323510 23600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:22.323621 23600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:22.323731 23600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:56:22.333460 23600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:45883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=34193
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:22.334688 23600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:22.335969 23600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:22.348920 23605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:22.348877 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:22.348923 23606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:22.349964 23600 server_base.cc:1034] running on GCE node
I20260927 22:56:22.350492 23600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:22.351374 23600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:22.352571 23600 hybrid_clock.cc:648] HybridClock initialized: now 1790549782352516 us; error 48 us; skew 500 ppm
I20260927 22:56:22.352837 23600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:22 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549782, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:22.355767 23600 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:56:22.357067 23600 webserver.cc:460] Webserver started at http://127.8.204.130:34193/ using document root <none> and password file <none>
I20260927 22:56:22.357697 23600 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:22.357810 23600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:22.363005 23600 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:22.365789 23615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:22.367314 23600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:56:22.367465 23600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "b5eeaf9486c04625b57af395ec03d498"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4b439a4e004846091849866532a65060"
server_key_iv: "7bfbb54373a96dfde42805d0fd8ab35d"
server_key_version: "encryptionkey@0"
I20260927 22:56:22.368404 23600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:22.399001 23600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:22.403646 23600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:22.403931 23600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:22.404951 23600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:22.406418 23600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:22.406500 23600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:22.406652 23600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:22.406741 23600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:22.438007 23600 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:45883
I20260927 22:56:22.438093 23727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:45883 every 8 connection(s)
I20260927 22:56:22.439459 23600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:22.440531 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23600
I20260927 22:56:22.441249 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:34967
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:40621
--webserver_port=41887
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:56:22 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549782, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:22.461335 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.442356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:44619 (local address 127.8.204.190:40621)
0927 22:56:22.442569 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.442575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.443885 (+ 1310us) server_negotiation.cc:407] Connection header received
0927 22:56:22.446777 (+ 2892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.446781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.446851 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.446938 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:22.449098 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.449839 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.451495 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.451955 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.452115 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.455541 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:22.455546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:22.455549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:22.455581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:22.458258 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.458977 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.458981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.458984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.459115 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.459591 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.459595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.459597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.459868 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:22.460034 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.460705 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.460937 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:56:22.462431 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.442864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:44619)
0927 22:56:22.443701 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:22.443754 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:22.446483 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:22.447234 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:22.447248 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:22.448047 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:22.448887 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.448917 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.450022 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.450030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.451340 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.451362 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.452200 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.452207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.452449 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.453127 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:22.453150 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:22.455316 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:22.458397 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:22.458406 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:22.458421 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:22.458838 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:22.459234 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:22.459242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:22.459246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:22.459458 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:22.460002 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:22.460010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:22.460328 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 22:56:22.460591 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 22:56:22.461112 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":354,"threads_started":1}
I20260927 22:56:22.463997 23728 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40621
I20260927 22:56:22.464324 23728 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:22.465011 23728 heartbeater.cc:511] Master 127.8.204.190:40621 requested a full tablet report, sending...
I20260927 22:56:22.466425 23411 ts_manager.cc:194] Registered new tserver with Master: b5eeaf9486c04625b57af395ec03d498 (127.8.204.130:45883)
I20260927 22:56:22.467257 23411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:44619
W20260927 22:56:22.772444 23731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:22.772722 23731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:22.772909 23731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:22.780193 23731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:22.780303 23731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:22.780499 23731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:56:22.790073 23731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:34967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=41887
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:22.791409 23731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:22.792690 23731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:22.805686 23736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:22.805722 23737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:22.805759 23739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:22.807389 23731 server_base.cc:1034] running on GCE node
I20260927 22:56:22.807971 23731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:22.808868 23731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:22.810091 23731 hybrid_clock.cc:648] HybridClock initialized: now 1790549782810044 us; error 58 us; skew 500 ppm
I20260927 22:56:22.810417 23731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:22 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549782, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:22.813593 23731 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:56:22.814947 23731 webserver.cc:460] Webserver started at http://127.8.204.131:41887/ using document root <none> and password file <none>
I20260927 22:56:22.815657 23731 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:22.815768 23731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:22.820950 23731 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:56:22.823572 23746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:22.824906 23731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:22.825047 23731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "61f5d07333094f3c84c20305302580b4"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4968dcd23542053ef315675d44a97e77"
server_key_iv: "f2c9b6e68fe997d36b494cd55f81a41b"
server_key_version: "encryptionkey@0"
I20260927 22:56:22.825975 23731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:22.849912 23731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:22.854306 23731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:22.854538 23731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:22.855473 23731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:22.856724 23731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:22.856807 23731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:22.856886 23731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:22.856940 23731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:22.889858 23731 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:34967
I20260927 22:56:22.889943 23858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:34967 every 8 connection(s)
I20260927 22:56:22.891361 23731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:22.899329 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23731
Sep 27 22:56:22 dist-test-slave-40vx krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549782, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:22.913662 23598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.893805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:48777 (local address 127.8.204.190:40621)
0927 22:56:22.893982 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.893987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.895296 (+ 1309us) server_negotiation.cc:407] Connection header received
0927 22:56:22.897306 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.897311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.897380 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.897471 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:22.898894 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.900056 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.902476 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.902891 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.902989 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.907177 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:22.907185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:22.907205 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:22.907250 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:22.910520 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.911249 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.911255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.911258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.911444 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.911896 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.911901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.911905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.912210 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:22.912372 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.913084 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.913318 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:56:22.914777 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.894221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:48777)
0927 22:56:22.895074 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:22.895128 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:22.897035 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:22.897691 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:22.897700 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:22.898176 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:22.898730 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.898747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.900235 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.900239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.902328 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.902347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.903127 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.903134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.903504 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.904231 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:22.904253 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:22.906961 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:22.910664 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:22.910674 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:22.910698 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:22.911117 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:22.911572 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:22.911577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:22.911581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:22.911775 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:22.912356 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:22.912364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:22.912712 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 22:56:22.912997 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 22:56:22.913586 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":307,"threads_started":1}
I20260927 22:56:22.916085 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.900236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40838 (local address 127.8.204.190:40621)
0927 22:56:22.900717 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.900722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.901212 (+ 490us) server_negotiation.cc:407] Connection header received
0927 22:56:22.901355 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.901358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.901413 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.901495 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:22.903370 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.904364 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.906097 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.906616 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.906756 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.910261 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:22.910265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:22.910269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:22.910307 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:22.912961 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.913654 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.913662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.913667 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.913838 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:22.914331 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:22.914336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:22.914339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:22.914603 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:22.914848 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.915453 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.915728 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260927 22:56:22.916388 23859 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40621
I20260927 22:56:22.916759 23859 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:22.917559 23859 heartbeater.cc:511] Master 127.8.204.190:40621 requested a full tablet report, sending...
I20260927 22:56:22.918967 23411 ts_manager.cc:194] Registered new tserver with Master: 61f5d07333094f3c84c20305302580b4 (127.8.204.131:34967)
I20260927 22:56:22.919888 23411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:48777
I20260927 22:56:22.931143 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:22.937789 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.932157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.8.204.190:40621)
0927 22:56:22.932402 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.932408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.932668 (+ 260us) server_negotiation.cc:407] Connection header received
0927 22:56:22.932830 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.932834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.932974 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.933125 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:22.933544 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.934245 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.935749 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.936226 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.936318 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.936711 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:22.937248 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:22.937351 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.937436 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.937495 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:56:22.938704 23411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:22.946033 23411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:22.968797 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.957842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.8.204.129:41957)
0927 22:56:22.960388 (+ 2546us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.960400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.960431 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:56:22.960536 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.960543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.960790 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.961018 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:22.962184 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.963904 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.966436 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.967649 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.967804 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.968017 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.968290 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.968449 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2382,"thread_start_us":2246,"threads_started":1}
I20260927 22:56:22.968868 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.957684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55346)
0927 22:56:22.959598 (+ 1914us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:22.959635 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:22.959815 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:22.961344 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:22.961354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:22.961405 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:22.961883 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.961900 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.964078 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.964083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.966232 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.966249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.967864 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.967874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.968067 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.968113 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:56:22.968249 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:56:22.968432 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1666,"mutex_wait_us":23,"thread_start_us":444,"threads_started":1}
I20260927 22:56:22.969475 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.957819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:34967 (local address 127.0.0.1:56812)
0927 22:56:22.960592 (+ 2773us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:22.960616 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:22.960756 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:22.961922 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:22.961929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:22.961951 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:22.962372 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.962383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.964543 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.964550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.966824 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.966844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.968726 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.968733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.968914 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.968949 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 22:56:22.969018 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:56:22.969147 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2625,"thread_start_us":1840,"threads_started":1}
I20260927 22:56:22.970563 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.958480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.8.204.131:34967)
0927 22:56:22.961356 (+ 2876us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.961365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.961399 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:56:22.961500 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.961508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.961720 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.962559 (+ 839us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:22.962851 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.964423 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.967035 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.968498 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.969739 (+ 1241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.969973 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.970048 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.970208 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2708,"thread_start_us":2544,"threads_started":1}
I20260927 22:56:22.971202 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.960007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:45883 (local address 127.0.0.1:46818)
0927 22:56:22.961672 (+ 1665us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:22.961695 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:22.961815 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:22.963885 (+ 2070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:22.963889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:22.963916 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:22.964308 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.964317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.966184 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.966189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.968360 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:22.968376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.970533 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.970539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:22.970721 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.970749 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:56:22.970841 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:56:22.970936 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":1,"thread_start_us":1368,"threads_started":1}
I20260927 22:56:22.971939 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:22.960639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46818 (local address 127.8.204.130:45883)
0927 22:56:22.963319 (+ 2680us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:22.963329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:22.963358 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:56:22.963476 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:22.963482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:22.963685 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:22.963869 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:22.964484 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.966016 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.968550 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:22.970067 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:22.970195 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:22.970366 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:22.971524 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0927 22:56:22.971690 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2484,"thread_start_us":2365,"threads_started":1}
I20260927 22:56:22.973299 23531 tablet_service.cc:1467] Processing CreateTablet for tablet a90fa8a678254bd6ad1b71bff4809b31 (DEFAULT_TABLE table=test-table [id=8ef721761faa40ef9d726a70f6b6bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:22.974212 23794 tablet_service.cc:1467] Processing CreateTablet for tablet a90fa8a678254bd6ad1b71bff4809b31 (DEFAULT_TABLE table=test-table [id=8ef721761faa40ef9d726a70f6b6bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:22.974826 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90fa8a678254bd6ad1b71bff4809b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:22.974968 23663 tablet_service.cc:1467] Processing CreateTablet for tablet a90fa8a678254bd6ad1b71bff4809b31 (DEFAULT_TABLE table=test-table [id=8ef721761faa40ef9d726a70f6b6bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:22.975540 23794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90fa8a678254bd6ad1b71bff4809b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:22.976136 23663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90fa8a678254bd6ad1b71bff4809b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:22.990698 23876 tablet_bootstrap.cc:492] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Bootstrap starting.
I20260927 22:56:22.992673 23875 tablet_bootstrap.cc:492] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Bootstrap starting.
I20260927 22:56:22.994261 23877 tablet_bootstrap.cc:492] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Bootstrap starting.
I20260927 22:56:22.995693 23876 tablet_bootstrap.cc:654] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:22.996289 23875 tablet_bootstrap.cc:654] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:22.996999 23876 log.cc:824] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:22.997185 23877 tablet_bootstrap.cc:654] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:22.997473 23875 log.cc:824] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:22.998412 23877 log.cc:824] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:23.000218 23876 tablet_bootstrap.cc:492] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: No bootstrap required, opened a new log
I20260927 22:56:23.000478 23876 ts_tablet_manager.cc:1397] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260927 22:56:23.001346 23877 tablet_bootstrap.cc:492] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: No bootstrap required, opened a new log
I20260927 22:56:23.001533 23877 ts_tablet_manager.cc:1397] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:56:23.001642 23875 tablet_bootstrap.cc:492] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: No bootstrap required, opened a new log
I20260927 22:56:23.001880 23875 ts_tablet_manager.cc:1397] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:56:23.008675 23877 raft_consensus.cc:348] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.009025 23877 raft_consensus.cc:374] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:23.009159 23877 raft_consensus.cc:729] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7aa8c7767df44538fbef2a33a1fd1b4, State: Initialized, Role: FOLLOWER
I20260927 22:56:23.009734 23877 consensus_queue.cc:260] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.011147 23876 raft_consensus.cc:348] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.011826 23876 raft_consensus.cc:374] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:23.011971 23876 raft_consensus.cc:729] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5eeaf9486c04625b57af395ec03d498, State: Initialized, Role: FOLLOWER
I20260927 22:56:23.012333 23596 heartbeater.cc:503] Master 127.8.204.190:40621 was elected leader, sending a full tablet report...
I20260927 22:56:23.012208 23875 raft_consensus.cc:348] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.012588 23875 raft_consensus.cc:374] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:23.012673 23875 raft_consensus.cc:729] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f5d07333094f3c84c20305302580b4, State: Initialized, Role: FOLLOWER
I20260927 22:56:23.012730 23876 consensus_queue.cc:260] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.013041 23877 ts_tablet_manager.cc:1428] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Time spent starting tablet: real 0.011s user 0.004s sys 0.009s
I20260927 22:56:23.013291 23875 consensus_queue.cc:260] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.014448 23859 heartbeater.cc:503] Master 127.8.204.190:40621 was elected leader, sending a full tablet report...
I20260927 22:56:23.015303 23875 ts_tablet_manager.cc:1428] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:56:23.016206 23728 heartbeater.cc:503] Master 127.8.204.190:40621 was elected leader, sending a full tablet report...
I20260927 22:56:23.016860 23876 ts_tablet_manager.cc:1428] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260927 22:56:23.127252 23882 raft_consensus.cc:482] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:23.127624 23882 raft_consensus.cc:504] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.129426 23882 leader_election.cc:288] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957), 61f5d07333094f3c84c20305302580b4 (127.8.204.131:34967)
I20260927 22:56:23.136385 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.130041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:43409 (local address 127.8.204.131:34967)
0927 22:56:23.130302 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.130310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.131056 (+ 746us) server_negotiation.cc:407] Connection header received
0927 22:56:23.131244 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.131248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.131361 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.131455 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:23.132055 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.132746 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.134577 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.135435 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.135619 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.135697 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.135994 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.136120 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:56:23.136436 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.129755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:52757 (local address 127.8.204.129:41957)
0927 22:56:23.129989 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.129999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.130949 (+ 950us) server_negotiation.cc:407] Connection header received
0927 22:56:23.131190 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.131196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.131257 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.131385 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:23.131993 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.132698 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.134608 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.135452 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.135619 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.135708 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.136033 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.136168 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:56:23.136605 23885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.129729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:34967 (local address 127.8.204.130:43409)
0927 22:56:23.130956 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.130970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:23.131077 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:23.131471 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:23.131475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:23.131506 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:23.131891 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:23.131900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.132883 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.132888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:23.134420 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:23.134432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.135664 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.135670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:23.135840 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.135863 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:56:23.135969 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:56:23.136104 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1124,"mutex_wait_us":2,"thread_start_us":894,"threads_started":1}
I20260927 22:56:23.136631 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.129660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.8.204.130:52757)
0927 22:56:23.130829 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.130844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:23.131043 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:23.131415 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:23.131419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:23.131443 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:23.131848 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:23.131860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.132823 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.132831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:23.134455 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:23.134467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.135745 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.135756 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:23.135883 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.135905 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:56:23.136007 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:56:23.136127 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":83,"thread_start_us":638,"threads_started":1}
I20260927 22:56:23.137791 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90fa8a678254bd6ad1b71bff4809b31" candidate_uuid: "b5eeaf9486c04625b57af395ec03d498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" is_pre_election: true
I20260927 22:56:23.137792 23814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90fa8a678254bd6ad1b71bff4809b31" candidate_uuid: "b5eeaf9486c04625b57af395ec03d498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f5d07333094f3c84c20305302580b4" is_pre_election: true
I20260927 22:56:23.138373 23551 raft_consensus.cc:2454] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5eeaf9486c04625b57af395ec03d498 in term 0.
I20260927 22:56:23.138373 23814 raft_consensus.cc:2454] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5eeaf9486c04625b57af395ec03d498 in term 0.
I20260927 22:56:23.139091 23617 leader_election.cc:302] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 61f5d07333094f3c84c20305302580b4, b5eeaf9486c04625b57af395ec03d498, e7aa8c7767df44538fbef2a33a1fd1b4; no voters:
I20260927 22:56:23.139539 23882 raft_consensus.cc:2781] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:23.139689 23882 raft_consensus.cc:482] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:23.139797 23882 raft_consensus.cc:3037] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:23.141971 23882 raft_consensus.cc:504] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.142586 23882 leader_election.cc:288] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [CANDIDATE]: Term 1 election: Requested vote from peers e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957), 61f5d07333094f3c84c20305302580b4 (127.8.204.131:34967)
W20260927 22:56:23.142949 23860 tablet.cc:2369] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:23.142971 23551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90fa8a678254bd6ad1b71bff4809b31" candidate_uuid: "b5eeaf9486c04625b57af395ec03d498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
I20260927 22:56:23.143136 23814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90fa8a678254bd6ad1b71bff4809b31" candidate_uuid: "b5eeaf9486c04625b57af395ec03d498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f5d07333094f3c84c20305302580b4"
I20260927 22:56:23.143177 23551 raft_consensus.cc:3037] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:23.143376 23814 raft_consensus.cc:3037] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:23.145537 23814 raft_consensus.cc:2454] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5eeaf9486c04625b57af395ec03d498 in term 1.
I20260927 22:56:23.145790 23551 raft_consensus.cc:2454] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5eeaf9486c04625b57af395ec03d498 in term 1.
I20260927 22:56:23.145946 23619 leader_election.cc:302] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f5d07333094f3c84c20305302580b4, b5eeaf9486c04625b57af395ec03d498; no voters:
I20260927 22:56:23.146277 23882 raft_consensus.cc:2781] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:23.146725 23882 raft_consensus.cc:686] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 1 LEADER]: Becoming Leader. State: Replica: b5eeaf9486c04625b57af395ec03d498, State: Running, Role: LEADER
I20260927 22:56:23.147199 23882 consensus_queue.cc:237] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } }
I20260927 22:56:23.151959 23409 catalog_manager.cc:5495] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5eeaf9486c04625b57af395ec03d498 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "b5eeaf9486c04625b57af395ec03d498" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5eeaf9486c04625b57af395ec03d498" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 45883 } health_report { overall_health: HEALTHY } } }
W20260927 22:56:23.191756 23729 tablet.cc:2369] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:23.209204 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.204163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.8.204.130:45883)
0927 22:56:23.204339 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.204344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.204360 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:56:23.204442 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.204446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.204520 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.204642 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:23.204965 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.205661 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.207202 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.207655 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.207747 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.208209 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:23.208730 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:23.208837 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.208923 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.208988 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:56:23.217443 23551 raft_consensus.cc:1261] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Refusing update from remote peer b5eeaf9486c04625b57af395ec03d498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:23.217443 23814 raft_consensus.cc:1261] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Refusing update from remote peer b5eeaf9486c04625b57af395ec03d498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:23.218431 23886 consensus_queue.cc:1035] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f5d07333094f3c84c20305302580b4" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:23.218767 23882 consensus_queue.cc:1035] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7aa8c7767df44538fbef2a33a1fd1b4" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:23.230751 23890 mvcc.cc:205] Tried to move back new op lower bound from 7334091912048652288 to 7334091911775494144. Current Snapshot: MvccSnapshot[applied={T|T < 7334091912048652288}]
I20260927 22:56:23.231236 23889 mvcc.cc:205] Tried to move back new op lower bound from 7334091912048652288 to 7334091911775494144. Current Snapshot: MvccSnapshot[applied={T|T < 7334091912048652288}]
I20260927 22:56:23.242034 23411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:23.242297 23411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:23.245515 23411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Sending DeleteTablet for 3 replicas of tablet a90fa8a678254bd6ad1b71bff4809b31
I20260927 22:56:23.246723 23531 tablet_service.cc:1514] Processing DeleteTablet for tablet a90fa8a678254bd6ad1b71bff4809b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:23 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:55346
I20260927 22:56:23.246977 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23448
I20260927 22:56:23.246981 23794 tablet_service.cc:1514] Processing DeleteTablet for tablet a90fa8a678254bd6ad1b71bff4809b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:23 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:56812
I20260927 22:56:23.247556 23663 tablet_service.cc:1514] Processing DeleteTablet for tablet a90fa8a678254bd6ad1b71bff4809b31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:23 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:46818
I20260927 22:56:23.248736 23900 tablet_replica.cc:330] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: stopping tablet replica
I20260927 22:56:23.250985 23900 raft_consensus.cc:2229] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:23.251724 23900 raft_consensus.cc:2258] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:23.254284 23900 ts_tablet_manager.cc:1905] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:23.259815 23901 tablet_replica.cc:330] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: stopping tablet replica
I20260927 22:56:23.260320 23901 raft_consensus.cc:2229] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:23.260895 23901 raft_consensus.cc:2258] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:23.261413 23900 ts_tablet_manager.cc:1918] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:23.261447 23899 tablet_replica.cc:330] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: stopping tablet replica
I20260927 22:56:23.261582 23900 log.cc:1196] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/a90fa8a678254bd6ad1b71bff4809b31
I20260927 22:56:23.261888 23899 raft_consensus.cc:2229] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:23.262233 23900 ts_tablet_manager.cc:1939] T a90fa8a678254bd6ad1b71bff4809b31 P b5eeaf9486c04625b57af395ec03d498: Deleting consensus metadata
I20260927 22:56:23.262427 23899 raft_consensus.cc:2258] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:23.263953 23901 ts_tablet_manager.cc:1905] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:23.264362 23398 catalog_manager.cc:4841] TS b5eeaf9486c04625b57af395ec03d498 (127.8.204.130:45883): tablet a90fa8a678254bd6ad1b71bff4809b31 (table test-table [id=8ef721761faa40ef9d726a70f6b6bb43]) successfully deleted
I20260927 22:56:23.268185 23899 ts_tablet_manager.cc:1905] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:23.270948 23901 ts_tablet_manager.cc:1918] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:23.271199 23901 log.cc:1196] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/a90fa8a678254bd6ad1b71bff4809b31
I20260927 22:56:23.271800 23901 ts_tablet_manager.cc:1939] T a90fa8a678254bd6ad1b71bff4809b31 P 61f5d07333094f3c84c20305302580b4: Deleting consensus metadata
I20260927 22:56:23.272895 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.248252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55352 (local address 127.8.204.129:41957)
0927 22:56:23.248463 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.248472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.248495 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:56:23.248565 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.248573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.248695 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.248826 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:23.250544 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.252119 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.254144 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.254976 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.255126 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.265453 (+ 10327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:23.265464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:23.265480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:23.265521 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:23.269414 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:23.270380 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:23.270389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:23.270393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:23.270538 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:23.270922 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:23.270926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:23.270929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:23.271626 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:23.271948 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.272236 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.272521 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:56:23.273638 23398 catalog_manager.cc:4841] TS 61f5d07333094f3c84c20305302580b4 (127.8.204.131:34967): tablet a90fa8a678254bd6ad1b71bff4809b31 (table test-table [id=8ef721761faa40ef9d726a70f6b6bb43]) successfully deleted
I20260927 22:56:23.273736 23591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:23.362869 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23448
W20260927 22:56:23.381109 23396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.129:41957 (error 108)
I20260927 22:56:23.381332 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23600
I20260927 22:56:23.383145 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.382641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55364)
0927 22:56:23.382850 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.382959 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 22:56:23.383574 23396 catalog_manager.cc:4584] TS e7aa8c7767df44538fbef2a33a1fd1b4 (127.8.204.129:41957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a90fa8a678254bd6ad1b71bff4809b31: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
I20260927 22:56:23.399834 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.382168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.8.204.130:45883)
0927 22:56:23.382359 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.382367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.382524 (+ 157us) server_negotiation.cc:407] Connection header received
0927 22:56:23.382734 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.382740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.382837 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.382942 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:23.384915 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.386115 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.387982 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.388616 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.388725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.392018 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:23.392035 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:23.392057 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:23.392103 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:23.396712 (+ 4609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:23.397559 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:23.397568 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:23.397572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:23.397722 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:23.398139 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:23.398144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:23.398147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:23.398536 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:23.398861 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.399199 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.399464 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:56:23.400509 23723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:23.437335 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.436877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55374)
0927 22:56:23.437071 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.437182 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:56:23.438364 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.437991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55380)
0927 22:56:23.438140 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.438252 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:56:23.498932 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.498357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55394)
0927 22:56:23.498630 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.498746 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:56:23.500135 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.499747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55410)
0927 22:56:23.499913 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.500003 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:56:23.515758 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23600
I20260927 22:56:23.534118 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23731
I20260927 22:56:23.551982 23872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.539295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.8.204.131:34967)
0927 22:56:23.539507 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:23.539514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:23.539638 (+ 124us) server_negotiation.cc:407] Connection header received
0927 22:56:23.539867 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:23.539877 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:23.539941 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:23.540048 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:23.541517 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.542201 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.543600 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:23.544010 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:23.544144 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:23.546644 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:23.546656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:23.546672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:23.546709 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:23.549270 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:23.549932 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:23.549939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:23.549942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:23.550081 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:23.550451 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:23.550457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:23.550460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:23.550800 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:23.551098 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:23.551419 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 22:56:23.551646 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:56:23.552604 23854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:23.580658 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.580073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55422)
0927 22:56:23.580316 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.580430 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 22:56:23.581763 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.581343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55438)
0927 22:56:23.581553 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.581634 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:56:23.642774 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23731
I20260927 22:56:23.658999 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23378
I20260927 22:56:23.660316 23441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:23.774775 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:23.773792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:41957 (local address 127.0.0.1:55452)
0927 22:56:23.774425 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:23.774548 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469}
I20260927 22:56:23.774773 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23378
I20260927 22:56:23.796344 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40621
--webserver_interface=127.8.204.190
--webserver_port=39063
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:24.123593 23907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:24.123883 23907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:24.124058 23907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:24.131302 23907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:24.131431 23907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:24.131515 23907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:24.131613 23907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:24.131677 23907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:24.141497 23907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=39063
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:24.142817 23907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:24.144129 23907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:24.155656 23912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:24.155682 23913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:24.155818 23915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:24.156504 23907 server_base.cc:1034] running on GCE node
I20260927 22:56:24.157291 23907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:24.158664 23907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:24.159876 23907 hybrid_clock.cc:648] HybridClock initialized: now 1790549784159836 us; error 35 us; skew 500 ppm
I20260927 22:56:24.160274 23907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:24 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549784, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:24.163374 23907 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:24.164822 23907 webserver.cc:460] Webserver started at http://127.8.204.190:39063/ using document root <none> and password file <none>
I20260927 22:56:24.165531 23907 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:24.165630 23907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:24.170859 23907 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:56:24.173548 23922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:24.174901 23907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:56:24.175079 23907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "a1e86bb2d76d4e32861fe89bece75632"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "dfa21dc5e19edfc1a446ddab91ef0e67"
server_key_iv: "cff4f9dce15dcc85c673465d3eaad740"
server_key_version: "encryptionkey@0"
I20260927 22:56:24.176142 23907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:24.193096 23907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:24.196902 23907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:24.197175 23907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:24.215552 23907 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40621
I20260927 22:56:24.215577 23973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40621 every 8 connection(s)
I20260927 22:56:24.217005 23907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:24.220474 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23907
I20260927 22:56:24.221290 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:41957
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:40621
--webserver_port=35017
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:56:24.223881 23974 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:56:24.227095 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap starting.
I20260927 22:56:24.257159 23974 log.cc:824] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:24.277721 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:56:24.278404 23974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632: Bootstrap complete.
I20260927 22:56:24.293416 23974 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:24.294651 23974 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e86bb2d76d4e32861fe89bece75632, State: Initialized, Role: FOLLOWER
I20260927 22:56:24.295430 23974 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:24.295686 23974 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:24.295830 23974 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:24.296011 23974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:56:24.299952 23974 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:24.300433 23974 leader_election.cc:302] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1e86bb2d76d4e32861fe89bece75632; no voters:
I20260927 22:56:24.301195 23974 leader_election.cc:288] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:56:24.301342 23978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:56:24.305356 23978 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [term 3 LEADER]: Becoming Leader. State: Replica: a1e86bb2d76d4e32861fe89bece75632, State: Running, Role: LEADER
I20260927 22:56:24.305943 23978 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } }
I20260927 22:56:24.306107 23974 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:24.311292 23979 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1e86bb2d76d4e32861fe89bece75632. Latest consensus state: current_term: 3 leader_uuid: "a1e86bb2d76d4e32861fe89bece75632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } } }
I20260927 22:56:24.312570 23979 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:24.315564 23980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a1e86bb2d76d4e32861fe89bece75632" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1e86bb2d76d4e32861fe89bece75632" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40621 } } }
I20260927 22:56:24.315960 23980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1e86bb2d76d4e32861fe89bece75632 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:24.322788 23987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:24.333951 23987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:24.335916 23987 catalog_manager.cc:1259] Loaded cluster ID: e4c396da7a49417b814e021ec65b8ec0
I20260927 22:56:24.336109 23987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:56:24.338711 23987 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa9982ab9ad google::LogMessage::Fail() at ??:0
@ 0x7fa9982ae88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa9982ab4de google::LogMessage::Flush() at ??:0
@ 0x7fa9982aca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa9adcde512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa9adcdb976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa9add63610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa9add63439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa99aef7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa99aefa91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa99aefa799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa99aed5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa99e1af6db start_thread at ??:0
@ 0x7fa99714561f clone at ??:0
W20260927 22:56:24.628731 23976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:24.629016 23976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:24.629199 23976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:24.636278 23976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:24.636400 23976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:24.636582 23976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:56:24.645947 23976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:41957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=35017
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:24.647292 23976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:24.648664 23976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:24.661659 24015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:24.661783 24016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:24.661783 24018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:24.662621 23976 server_base.cc:1034] running on GCE node
I20260927 22:56:24.663236 23976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:24.664191 23976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:24.665416 23976 hybrid_clock.cc:648] HybridClock initialized: now 1790549784665380 us; error 48 us; skew 500 ppm
I20260927 22:56:24.665730 23976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:24 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549784, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:24.668967 23976 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:56:24.670349 23976 webserver.cc:460] Webserver started at http://127.8.204.129:35017/ using document root <none> and password file <none>
I20260927 22:56:24.671034 23976 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:24.671181 23976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:24.676628 23976 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:56:24.679665 24025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:24.680967 23976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:56:24.681133 23976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "e7aa8c7767df44538fbef2a33a1fd1b4"
format_stamp: "Formatted at 2026-09-27 22:56:18 on dist-test-slave-40vx"
server_key: "31e574436ac3f186b2f1bda9baf20a6f"
server_key_iv: "3c65d292539c713dea8b64b73413467c"
server_key_version: "encryptionkey@0"
I20260927 22:56:24.682073 23976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:24.714305 23976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:24.718493 23976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:24.718778 23976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:24.719851 23976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:24.721681 24032 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 22:56:24.726948 24032 ts_tablet_manager.cc:1852] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 22:56:24.727996 24032 ts_tablet_manager.cc:1905] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:24.734783 24032 ts_tablet_manager.cc:1918] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:24.734947 24032 log.cc:1196] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/a90fa8a678254bd6ad1b71bff4809b31
I20260927 22:56:24.735399 24032 ts_tablet_manager.cc:1939] T a90fa8a678254bd6ad1b71bff4809b31 P e7aa8c7767df44538fbef2a33a1fd1b4: Deleting consensus metadata
I20260927 22:56:24.735766 23976 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 22:56:24.735900 23976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260927 22:56:24.736027 23976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:24.736112 23976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:24.769452 23976 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:41957
I20260927 22:56:24.769538 24138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:41957 every 8 connection(s)
I20260927 22:56:24.770761 23976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:24.775696 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:24.773625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:53431)
0927 22:56:24.774410 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:24.774658 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":298,"threads_started":1}
W20260927 22:56:24.776333 24139 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:40621 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.190:40621: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
I20260927 22:56:24.777107 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:24.776809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:45459)
0927 22:56:24.776941 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:24.777009 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 22:56:24.778060 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 23976
I20260927 22:56:24.778090 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:24.777737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:52863)
0927 22:56:24.777887 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:24.777970 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:56:24.778329 24139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:56:24.778736 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:45883
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:40621
--webserver_port=34193
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:25.106217 24142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:25.106496 24142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:25.106591 24142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:25.113794 24142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:25.113905 24142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:25.114044 24142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:56:25.123534 24142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:45883
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=34193
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:25.124769 24142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:25.126039 24142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:25.139103 24148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:25.139161 24150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:25.139184 24147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:25.140717 24142 server_base.cc:1034] running on GCE node
I20260927 22:56:25.141312 24142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:25.142258 24142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:25.143460 24142 hybrid_clock.cc:648] HybridClock initialized: now 1790549785143420 us; error 33 us; skew 500 ppm
I20260927 22:56:25.143751 24142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:25 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549785, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:25.146770 24142 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:56:25.148245 24142 webserver.cc:460] Webserver started at http://127.8.204.130:34193/ using document root <none> and password file <none>
I20260927 22:56:25.148927 24142 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:25.149035 24142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:25.154245 24142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:56:25.156849 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.158193 24142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:56:25.158370 24142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "b5eeaf9486c04625b57af395ec03d498"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4b439a4e004846091849866532a65060"
server_key_iv: "7bfbb54373a96dfde42805d0fd8ab35d"
server_key_version: "encryptionkey@0"
I20260927 22:56:25.159353 24142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:25.170849 24142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:25.174844 24142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:25.175113 24142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:25.176146 24142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:25.177449 24142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:25.177567 24142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.177706 24142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:25.177802 24142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.210505 24142 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:45883
I20260927 22:56:25.210541 24269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:45883 every 8 connection(s)
I20260927 22:56:25.212059 24142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:25.216974 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.215017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:40793)
0927 22:56:25.215816 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.216099 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":349,"threads_started":1}
W20260927 22:56:25.217737 24270 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:40621 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.190:40621: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
I20260927 22:56:25.218520 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.218215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:56241)
0927 22:56:25.218350 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.218417 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 22:56:25.219326 24272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.218979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:41011)
0927 22:56:25.219104 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.219175 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:56:25.219585 24270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:56:25.220863 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 24142
I20260927 22:56:25.221419 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:34967
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:40621
--webserver_port=41887
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:35511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:25.553241 24273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:25.553514 24273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:25.553669 24273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:25.561098 24273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:25.561216 24273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:25.561408 24273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:56:25.571470 24273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:35511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:34967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=41887
--tserver_master_addrs=127.8.204.190:40621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:25.572762 24273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:25.574113 24273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:25.587313 24281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:25.587426 24278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:25.587335 24279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:25.588212 24273 server_base.cc:1034] running on GCE node
I20260927 22:56:25.588809 24273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:25.589709 24273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:25.590930 24273 hybrid_clock.cc:648] HybridClock initialized: now 1790549785590877 us; error 63 us; skew 500 ppm
I20260927 22:56:25.591233 24273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:25 dist-test-slave-40vx krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549785, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:25.594271 24273 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:56:25.595749 24273 webserver.cc:460] Webserver started at http://127.8.204.131:41887/ using document root <none> and password file <none>
I20260927 22:56:25.596366 24273 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:25.596465 24273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:25.601431 24273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:56:25.604077 24288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.605321 24273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:25.605448 24273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "61f5d07333094f3c84c20305302580b4"
format_stamp: "Formatted at 2026-09-27 22:56:19 on dist-test-slave-40vx"
server_key: "4968dcd23542053ef315675d44a97e77"
server_key_iv: "f2c9b6e68fe997d36b494cd55f81a41b"
server_key_version: "encryptionkey@0"
I20260927 22:56:25.606454 24273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:25.630575 24273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:25.634277 24273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:25.634492 24273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:25.635409 24273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:25.636648 24273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:25.636737 24273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.636806 24273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:25.636859 24273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:25.669365 24273 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:34967
I20260927 22:56:25.669456 24400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:34967 every 8 connection(s)
I20260927 22:56:25.670894 24273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:25.673031 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 24273
I20260927 22:56:25.676595 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.674346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:47915)
0927 22:56:25.675257 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.675671 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":402,"threads_started":1}
W20260927 22:56:25.677374 24401 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:40621 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.190:40621: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
I20260927 22:56:25.678292 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.677997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:38975)
0927 22:56:25.678123 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.678189 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 22:56:25.679104 24404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.678774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:58531)
0927 22:56:25.678914 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.678999 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:56:25.679363 24401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:56:25.779682 24405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:25.778858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:55563)
0927 22:56:25.779343 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:25.779457 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":208,"threads_started":1}
I20260927 22:56:26.220759 24406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:26.220049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:58437)
0927 22:56:26.220460 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:26.220534 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":202,"threads_started":1}
I20260927 22:56:26.680572 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:26.679825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:59497)
0927 22:56:26.680271 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:26.680353 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":217,"threads_started":1}
I20260927 22:56:26.781129 24408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:26.780492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:48705)
0927 22:56:26.780900 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:26.780977 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":220,"threads_started":1}
I20260927 22:56:27.222324 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:27.221603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:34063)
0927 22:56:27.222053 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:27.222157 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260927 22:56:27.682157 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:27.681439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:43655)
0927 22:56:27.681901 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:27.681988 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":229,"threads_started":1}
I20260927 22:56:27.782639 24411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:27.781949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:48837)
0927 22:56:27.782368 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:27.782451 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":212,"threads_started":1}
I20260927 22:56:28.223991 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:28.223230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:49877)
0927 22:56:28.223734 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:28.223817 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":290,"threads_started":1}
I20260927 22:56:28.683877 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:28.683132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:48899)
0927 22:56:28.683624 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:28.683702 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":279,"threads_started":1}
I20260927 22:56:28.784284 24414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:28.783613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:43201)
0927 22:56:28.784047 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:28.784129 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":214,"threads_started":1}
I20260927 22:56:29.225703 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:29.224976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:36299)
0927 22:56:29.225411 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:29.225494 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":233,"threads_started":1}
I20260927 22:56:29.685642 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:29.684744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:42653)
0927 22:56:29.685334 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:29.685426 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":338,"threads_started":1}
I20260927 22:56:29.785825 24417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:29.785110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:35845)
0927 22:56:29.785581 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:29.785659 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":253,"threads_started":1}
I20260927 22:56:30.227418 24418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:30.226619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:38071)
0927 22:56:30.227082 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:30.227183 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260927 22:56:30.687459 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:30.686568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:54001)
0927 22:56:30.687128 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:30.687229 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
I20260927 22:56:30.787395 24420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:30.786631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:54829)
0927 22:56:30.787098 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:30.787177 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260927 22:56:31.228977 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:31.228294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:57025)
0927 22:56:31.228713 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:31.228810 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":210,"threads_started":1}
I20260927 22:56:31.689049 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:31.688367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:48575)
0927 22:56:31.688794 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:31.688887 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":219,"threads_started":1}
I20260927 22:56:31.789059 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:31.788276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:55171)
0927 22:56:31.788758 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:31.788849 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":228,"threads_started":1}
I20260927 22:56:32.230636 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:32.229908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:44845)
0927 22:56:32.230356 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:32.230432 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":204,"threads_started":1}
I20260927 22:56:32.690919 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:32.690014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:39629)
0927 22:56:32.690608 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:32.690695 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":293,"threads_started":1}
I20260927 22:56:32.790748 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:32.789961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:36459)
0927 22:56:32.790495 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:32.790582 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":282,"threads_started":1}
I20260927 22:56:33.232329 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:33.231610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:47495)
0927 22:56:33.232079 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:33.232169 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":229,"threads_started":1}
I20260927 22:56:33.692585 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:33.691903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:50149)
0927 22:56:33.692353 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:33.692431 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":214,"threads_started":1}
I20260927 22:56:33.792459 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:33.791676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:40225)
0927 22:56:33.792159 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:33.792294 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260927 22:56:34.233963 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:34.233241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:41917)
0927 22:56:34.233696 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:34.233789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":216,"threads_started":1}
I20260927 22:56:34.694134 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:34.693427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:37871)
0927 22:56:34.693890 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:34.693966 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":229,"threads_started":1}
I20260927 22:56:34.794061 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:34.793366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:34069)
0927 22:56:34.793814 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:34.793900 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260927 22:56:35.235816 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:35.235051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:37981)
0927 22:56:35.235557 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:35.235640 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":296,"threads_started":1}
I20260927 22:56:35.695689 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:35.694997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:50701)
0927 22:56:35.695451 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:35.695529 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":224,"threads_started":1}
I20260927 22:56:35.795697 24435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:35.794949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:55251)
0927 22:56:35.795429 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:35.795509 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":254,"threads_started":1}
I20260927 22:56:36.237675 24436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:36.236717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:42067)
0927 22:56:36.237318 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:36.237400 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":354,"threads_started":1}
I20260927 22:56:36.697633 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:36.696622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:37079)
0927 22:56:36.697089 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:36.697185 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":238,"threads_started":1}
I20260927 22:56:36.797441 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:36.796622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:36457)
0927 22:56:36.797141 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:36.797234 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":308,"threads_started":1}
I20260927 22:56:37.239243 24439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:37.238586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:32795)
0927 22:56:37.238997 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:37.239081 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260927 22:56:37.699517 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:37.698696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:55103)
0927 22:56:37.699202 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:37.699358 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":259,"threads_started":1}
I20260927 22:56:37.799191 24441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:37.798418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:36185)
0927 22:56:37.798910 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:37.799007 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":242,"threads_started":1}
I20260927 22:56:38.240967 24442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:38.240188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:51755)
0927 22:56:38.240675 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:38.240792 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":240,"threads_started":1}
I20260927 22:56:38.701123 24443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:38.700418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:41917)
0927 22:56:38.700854 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:38.700943 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":204,"threads_started":1}
I20260927 22:56:38.800856 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:38.800130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:39581)
0927 22:56:38.800601 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:38.800680 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":264,"threads_started":1}
I20260927 22:56:39.242555 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:39.241846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:42879)
0927 22:56:39.242277 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:39.242381 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260927 22:56:39.702834 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:39.702017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:60389)
0927 22:56:39.702498 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:39.702635 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":263,"threads_started":1}
I20260927 22:56:39.802564 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:39.801831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.129:54841)
0927 22:56:39.802307 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:39.802393 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260927 22:56:40.244406 24448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.243692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.130:42475)
0927 22:56:40.244120 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:40.244207 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260927 22:56:40.684139 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 23976
I20260927 22:56:40.704862 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.703941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40621 (local address 127.8.204.131:44577)
0927 22:56:40.704467 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:40.704607 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":257,"threads_started":1}
I20260927 22:56:40.707309 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.685278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.8.204.129:41957)
0927 22:56:40.685866 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:40.685884 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:40.685929 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:56:40.688509 (+ 2580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:40.688518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:40.688995 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:40.689361 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:40.690864 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.692143 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.693395 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.694088 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.694283 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:40.701024 (+ 6741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:40.701033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:40.701050 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:40.701089 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:40.703935 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:40.704830 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:40.704841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:40.704853 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:40.705091 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:40.705463 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:40.705466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:40.705468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:40.705990 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:40.706204 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:40.706552 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:56:40.706977 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":273,"threads_started":1}
I20260927 22:56:40.708720 24134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:40.793663 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 23976
I20260927 22:56:40.810523 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 24142
I20260927 22:56:40.828044 24452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.811528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.8.204.130:45883)
0927 22:56:40.812037 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:40.812050 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:40.812083 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:56:40.813647 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:40.813656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:40.814093 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:40.814483 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:40.816161 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.817358 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.818610 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.819312 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.819549 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:40.822023 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:40.822030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:40.822044 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:40.822078 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:40.824817 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:40.825486 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:40.825492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:40.825498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:40.825659 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:40.826033 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:40.826039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:40.826042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:40.826560 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:40.826875 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:40.827365 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 22:56:40.827767 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":251,"threads_started":1}
I20260927 22:56:40.829409 24265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:40.913844 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 24142
I20260927 22:56:40.931519 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 24273
I20260927 22:56:40.950999 24454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:40.932534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.8.204.131:34967)
0927 22:56:40.933152 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:40.933165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:40.933216 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:56:40.934863 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:40.934874 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:40.935422 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:40.935811 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:40.937311 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.938574 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.939974 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:40.940710 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:40.940931 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:40.943938 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:40.943951 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:40.943973 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:40.944066 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:40.947404 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:40.948188 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:40.948197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:40.948214 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:40.948395 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:40.948793 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:40.948798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:40.948802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:40.949389 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:40.949721 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:40.950230 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:56:40.950693 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":312,"threads_started":1}
I20260927 22:56:40.952548 24396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:41.037927 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 24273
2026-09-27T22:56:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24541 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:56:41 dist-test-slave-40vx krb5kdc[24458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:56:41 dist-test-slave-40vx krb5kdc[24458](info): set up 2 sockets
Sep 27 22:56:41 dist-test-slave-40vx krb5kdc[24458](info): commencing operation
krb5kdc: starting...
W20260927 22:56:42.139940 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:56:42 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:56:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:56:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:56:42.370362 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40233
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:42.691866 24471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:42.692195 24471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:42.692378 24471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:42.699555 24471 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:56:42.699705 24471 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:56:42.699775 24471 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:56:42.699853 24471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:56:42.699934 24471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:42.700048 24471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:42.700142 24471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:56:42.700223 24471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:56:42.710364 24471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:42.711730 24471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:42.713069 24471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:42.725396 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:42.725368 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:42.725426 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:42.726035 24471 server_base.cc:1034] running on GCE node
I20260927 22:56:42.726788 24471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:42.728238 24471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:42.729490 24471 hybrid_clock.cc:648] HybridClock initialized: now 1790549802729440 us; error 30 us; skew 500 ppm
I20260927 22:56:42.729807 24471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:42 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:42.732980 24471 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:56:42.734380 24471 webserver.cc:460] Webserver started at http://127.8.204.190:42221/ using document root <none> and password file <none>
I20260927 22:56:42.734997 24471 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:42.735080 24471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:42.735414 24471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:42.738231 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "4e8d418a13ba4357a6fecc6e59cd34d2"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-40vx"
server_key: "aa7da9650d2b617641b8720f5d165917"
server_key_iv: "fc983e749df514e9c2995a32358a2a4b"
server_key_version: "encryptionkey@0"
I20260927 22:56:42.738950 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "4e8d418a13ba4357a6fecc6e59cd34d2"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-40vx"
server_key: "aa7da9650d2b617641b8720f5d165917"
server_key_iv: "fc983e749df514e9c2995a32358a2a4b"
server_key_version: "encryptionkey@0"
I20260927 22:56:42.743854 24471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:56:42.747116 24486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:42.748472 24471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:56:42.748603 24471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "4e8d418a13ba4357a6fecc6e59cd34d2"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-40vx"
server_key: "aa7da9650d2b617641b8720f5d165917"
server_key_iv: "fc983e749df514e9c2995a32358a2a4b"
server_key_version: "encryptionkey@0"
I20260927 22:56:42.748771 24471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:42.764750 24471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:42.768908 24471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:42.769106 24471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:42.787294 24537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40233 every 8 connection(s)
I20260927 22:56:42.787308 24471 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40233
I20260927 22:56:42.788734 24471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:56:42.792508 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 24471
I20260927 22:56:42.792542 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:42.792702 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:56:42.793382 9010 external_mini_cluster.cc:1428] Setting key 8057834f27014b5c6b925825773c733d
I20260927 22:56:42.806421 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap starting.
Sep 27 22:56:42 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:42.810117 24538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:42.811170 24538 log.cc:824] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:42.814307 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: No bootstrap required, opened a new log
I20260927 22:56:42.818001 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:42.795391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.8.204.190:40233)
0927 22:56:42.796224 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:42.796236 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:42.796281 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:56:42.798434 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:42.798458 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:42.798872 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:42.799193 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:42.800946 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:42.802222 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:42.804143 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:42.804860 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:42.804989 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:42.809864 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:42.809874 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:42.809914 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:42.809961 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:42.813602 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:42.814258 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:42.814268 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:42.814278 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:42.814536 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:42.814903 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:42.814908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:42.814911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:42.815570 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:42.815903 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:42.816208 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:56:42.816655 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":614,"thread_start_us":409,"threads_started":1}
I20260927 22:56:42.823061 24538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:56:42.823443 24538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:42.823518 24538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8d418a13ba4357a6fecc6e59cd34d2, State: Initialized, Role: FOLLOWER
I20260927 22:56:42.823973 24538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:56:42.824159 24538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:56:42.824252 24538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:56:42.824375 24538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:42.826447 24538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:56:42.826959 24538 leader_election.cc:302] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e8d418a13ba4357a6fecc6e59cd34d2; no voters:
I20260927 22:56:42.827595 24538 leader_election.cc:288] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:56:42.827716 24543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:42.828408 24543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e8d418a13ba4357a6fecc6e59cd34d2, State: Running, Role: LEADER
I20260927 22:56:42.828944 24543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:56:42.829769 24538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:56:42.831601 24545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e8d418a13ba4357a6fecc6e59cd34d2. Latest consensus state: current_term: 1 leader_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } } }
I20260927 22:56:42.832015 24545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:42.832247 24544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } } }
I20260927 22:56:42.832631 24544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:56:42.834694 24549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:56:42.839794 24549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:56:42.849064 24549 catalog_manager.cc:1347] Generated new cluster ID: fe8a013763e94d51baf95747422a06b7
I20260927 22:56:42.849186 24549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:56:42.872352 24549 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:56:42.874032 24549 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:56:42.885553 24549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Generated new TSK 0
I20260927 22:56:42.886371 24549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:56:42.951345 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40233
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:43.283851 24564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:43.284185 24564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:43.284359 24564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:43.291419 24564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:43.291577 24564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:43.291770 24564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:56:43.301295 24564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:43.302587 24564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:43.303916 24564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:43.317353 24570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:43.317325 24572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:43.317377 24569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:43.318575 24564 server_base.cc:1034] running on GCE node
I20260927 22:56:43.319311 24564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:43.320286 24564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:43.321484 24564 hybrid_clock.cc:648] HybridClock initialized: now 1790549803321432 us; error 53 us; skew 500 ppm
I20260927 22:56:43.321780 24564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:43 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:43.324928 24564 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:56:43.326395 24564 webserver.cc:460] Webserver started at http://127.8.204.129:37797/ using document root <none> and password file <none>
I20260927 22:56:43.327028 24564 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:43.327150 24564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:43.327484 24564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:43.330432 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "ee7ba1b7db0548109c69b9ad8416cbe2"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "47bc583ed2fb1b336e3263c7137fdd41"
server_key_iv: "e4c71591938d564957bdd1bcd8f438c3"
server_key_version: "encryptionkey@0"
I20260927 22:56:43.331152 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "ee7ba1b7db0548109c69b9ad8416cbe2"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "47bc583ed2fb1b336e3263c7137fdd41"
server_key_iv: "e4c71591938d564957bdd1bcd8f438c3"
server_key_version: "encryptionkey@0"
I20260927 22:56:43.336757 24564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:56:43.340140 24579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:43.341475 24564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:43.341639 24564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "ee7ba1b7db0548109c69b9ad8416cbe2"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "47bc583ed2fb1b336e3263c7137fdd41"
server_key_iv: "e4c71591938d564957bdd1bcd8f438c3"
server_key_version: "encryptionkey@0"
I20260927 22:56:43.341854 24564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:43.369629 24564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:43.374063 24564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:43.374363 24564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:43.375375 24564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:43.376742 24564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:43.376814 24564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:43.376986 24564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:43.377048 24564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:43.409312 24564 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:40025
I20260927 22:56:43.409365 24691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:40025 every 8 connection(s)
I20260927 22:56:43.410645 24564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:56:43.416821 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 24564
I20260927 22:56:43.417061 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:56:43.417716 9010 external_mini_cluster.cc:1428] Setting key 6d967214f8d13119441849ed3955f76b
Sep 27 22:56:43 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:43.430948 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:43.413339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:52243 (local address 127.8.204.190:40233)
0927 22:56:43.413811 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:43.413818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:43.414810 (+ 992us) server_negotiation.cc:407] Connection header received
0927 22:56:43.416810 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:43.416814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:43.416880 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:43.417007 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:43.418653 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.419777 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.421357 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.421755 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.421887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:43.425258 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:43.425265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:43.425270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:43.425307 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:43.427948 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:43.428648 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:43.428654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:43.428657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:43.428783 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:43.429194 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:43.429201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:43.429204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:43.429519 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:43.429671 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:43.430353 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 22:56:43.430581 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260927 22:56:43.431998 24695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:43.413927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:52243)
0927 22:56:43.414626 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:43.414663 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:43.416480 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:43.417262 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:43.417271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:43.417771 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:43.418466 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:43.418495 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.419925 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.419931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:43.421196 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:43.421212 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.421970 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.421979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:43.422229 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:43.422915 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:43.422942 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:43.425051 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:43.428082 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:43.428091 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:43.428106 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:43.428523 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:43.428891 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:43.428895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:43.428899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:43.429084 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:43.429639 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:43.429647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:43.429986 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 22:56:43.430268 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:56:43.430806 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":260,"threads_started":1}
I20260927 22:56:43.433625 24692 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40233
I20260927 22:56:43.433966 24692 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:43.434893 24692 heartbeater.cc:511] Master 127.8.204.190:40233 requested a full tablet report, sending...
I20260927 22:56:43.437212 24503 ts_manager.cc:194] Registered new tserver with Master: ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129:40025)
I20260927 22:56:43.438663 24503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:52243
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:56:43.470166 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40233
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:43.796648 24698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:43.797036 24698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:43.797205 24698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:43.805326 24698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:43.805433 24698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:43.805639 24698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:56:43.818027 24698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:43.819365 24698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:43.820624 24698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:43.833665 24706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:43.833699 24703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:43.834209 24704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:43.834467 24698 server_base.cc:1034] running on GCE node
I20260927 22:56:43.835119 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:43.836223 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:43.837441 24698 hybrid_clock.cc:648] HybridClock initialized: now 1790549803837385 us; error 54 us; skew 500 ppm
I20260927 22:56:43.837751 24698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:43 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:43.840934 24698 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:56:43.842299 24698 webserver.cc:460] Webserver started at http://127.8.204.130:41895/ using document root <none> and password file <none>
I20260927 22:56:43.842939 24698 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:43.843039 24698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:43.843381 24698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:43.846477 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "2e820f2f6cb0446785d9be5325ee63a3"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "df0d6584e054d3b3833b34de46814b5d"
server_key_iv: "16b114eb990434f51c2c8a54fac188be"
server_key_version: "encryptionkey@0"
I20260927 22:56:43.847231 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "2e820f2f6cb0446785d9be5325ee63a3"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "df0d6584e054d3b3833b34de46814b5d"
server_key_iv: "16b114eb990434f51c2c8a54fac188be"
server_key_version: "encryptionkey@0"
I20260927 22:56:43.852664 24698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:56:43.856212 24713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:43.857579 24698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:56:43.857750 24698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "2e820f2f6cb0446785d9be5325ee63a3"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "df0d6584e054d3b3833b34de46814b5d"
server_key_iv: "16b114eb990434f51c2c8a54fac188be"
server_key_version: "encryptionkey@0"
I20260927 22:56:43.857967 24698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:43.871989 24698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:43.876228 24698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:43.876529 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:43.877463 24698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:43.878710 24698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:43.878782 24698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:43.878950 24698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:43.879002 24698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:43.911082 24698 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38431
I20260927 22:56:43.911124 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38431 every 8 connection(s)
I20260927 22:56:43.912523 24698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:56:43.913738 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 24698
I20260927 22:56:43.913930 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:56:43.914494 9010 external_mini_cluster.cc:1428] Setting key f5274faeca7ef999a9111ef46cab6177
Sep 27 22:56:43 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549803, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:43.936131 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:43.915513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:60497 (local address 127.8.204.190:40233)
0927 22:56:43.915731 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:43.915737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:43.917110 (+ 1373us) server_negotiation.cc:407] Connection header received
0927 22:56:43.920049 (+ 2939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:43.920056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:43.920141 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:43.920232 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:43.922654 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.923396 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.925973 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.926391 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.926531 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:43.930082 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:43.930087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:43.930090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:43.930127 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:43.932855 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:43.933556 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:43.933563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:43.933566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:43.933696 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:43.934197 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:43.934208 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:43.934210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:43.934456 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:43.934613 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:43.935427 (+ 814us) server_negotiation.cc:299] Negotiation successful
0927 22:56:43.935707 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:56:43.936954 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:43.916085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:60497)
0927 22:56:43.916919 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:43.916980 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:43.919732 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:43.920484 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:43.920498 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:43.921460 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:43.922462 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:43.922491 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.923550 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.923558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:43.925799 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:43.925825 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:43.926626 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:43.926634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:43.926856 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:43.927627 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:43.927651 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:43.929854 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:43.933015 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:43.933023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:43.933039 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:43.933441 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:43.933825 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:43.933831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:43.933834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:43.934064 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:43.934597 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:43.934604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:43.934980 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 22:56:43.935287 (+ 307us) client_negotiation.cc:240] Negotiation successful
0927 22:56:43.935792 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
I20260927 22:56:43.938716 24826 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40233
I20260927 22:56:43.939043 24826 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
I20260927 22:56:43.939911 24826 heartbeater.cc:511] Master 127.8.204.190:40233 requested a full tablet report, sending...
I20260927 22:56:43.941295 24503 ts_manager.cc:194] Registered new tserver with Master: 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
I20260927 22:56:43.942144 24503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:60497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:56:43.968277 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40233
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:56:44.290077 24831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:56:44.290354 24831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:56:44.290527 24831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:56:44.297578 24831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:56:44.297680 24831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:56:44.297900 24831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:56:44.307251 24831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:56:44.308517 24831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:56:44.309796 24831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:56:44.322579 24837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:44.322635 24836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:56:44.322551 24839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:56:44.323704 24831 server_base.cc:1034] running on GCE node
I20260927 22:56:44.324324 24831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:56:44.325225 24831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:56:44.326462 24831 hybrid_clock.cc:648] HybridClock initialized: now 1790549804326402 us; error 50 us; skew 500 ppm
I20260927 22:56:44.326781 24831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:56:44 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549804, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:56:44.330062 24831 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:56:44.331574 24831 webserver.cc:460] Webserver started at http://127.8.204.131:44501/ using document root <none> and password file <none>
I20260927 22:56:44.332188 24831 fs_manager.cc:359] Metadata directory not provided
I20260927 22:56:44.332285 24831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:56:44.332544 24831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:56:44.335441 24831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "bea7c04cfea94b71a8e499d18633a73d"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-40vx"
server_key: "99923efcc04589b31e68157f4c138e1a"
server_key_iv: "728f0dc0f02be10c3bbde35aa8bd96b6"
server_key_version: "encryptionkey@0"
I20260927 22:56:44.336171 24831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "bea7c04cfea94b71a8e499d18633a73d"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-40vx"
server_key: "99923efcc04589b31e68157f4c138e1a"
server_key_iv: "728f0dc0f02be10c3bbde35aa8bd96b6"
server_key_version: "encryptionkey@0"
I20260927 22:56:44.341702 24831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:56:44.346637 24846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:44.348467 24831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:56:44.348644 24831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "bea7c04cfea94b71a8e499d18633a73d"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-40vx"
server_key: "99923efcc04589b31e68157f4c138e1a"
server_key_iv: "728f0dc0f02be10c3bbde35aa8bd96b6"
server_key_version: "encryptionkey@0"
I20260927 22:56:44.348872 24831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:56:44.371496 24831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:56:44.375735 24831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:56:44.375985 24831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:56:44.376948 24831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:56:44.378311 24831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:56:44.378391 24831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:44.378476 24831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:56:44.378520 24831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:56:44.410048 24831 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:39549
I20260927 22:56:44.410127 24958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:39549 every 8 connection(s)
I20260927 22:56:44.411329 24831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:56:44.420411 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 24831
I20260927 22:56:44.420583 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:56:44.420994 9010 external_mini_cluster.cc:1428] Setting key b3b814d6ea6fa39934423f556639a430
Sep 27 22:56:44 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549804, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:56:44.431136 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.413954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:41137 (local address 127.8.204.190:40233)
0927 22:56:44.414159 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.414166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.415313 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 22:56:44.417131 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.417137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.417253 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.417359 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:44.418786 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.419619 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.421258 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.421791 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.421940 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.425689 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:44.425694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:44.425697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:44.425730 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:44.428277 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.429073 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.429077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.429079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.429211 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.429589 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.429592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.429605 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.429930 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:44.430050 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.430654 (+ 604us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.430816 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:56:44.431993 24961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.414442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:41137)
0927 22:56:44.415133 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.415168 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:44.416860 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:44.417571 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:44.417579 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:44.418063 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:56:44.418607 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.418628 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.419769 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.419793 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.421121 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.421133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.422014 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.422024 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.422272 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.423038 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:56:44.423066 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:56:44.425491 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:56:44.428400 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:44.428406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:44.428417 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:44.428968 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:44.429308 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:56:44.429311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:56:44.429313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:56:44.429490 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:56:44.430043 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:56:44.430049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:56:44.430320 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 22:56:44.430544 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 22:56:44.430898 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":287,"threads_started":1}
I20260927 22:56:44.433328 24959 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40233
I20260927 22:56:44.433619 24959 heartbeater.cc:464] Registering TS with master...
I20260927 22:56:44.434262 24959 heartbeater.cc:511] Master 127.8.204.190:40233 requested a full tablet report, sending...
I20260927 22:56:44.435912 24503 ts_manager.cc:194] Registered new tserver with Master: bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549)
I20260927 22:56:44.436699 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:56:44.436872 24503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:41137
I20260927 22:56:44.441679 24692 heartbeater.cc:503] Master 127.8.204.190:40233 was elected leader, sending a full tablet report...
I20260927 22:56:44.454141 24694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.441246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.8.204.190:40233)
0927 22:56:44.441425 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.441429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.441882 (+ 453us) server_negotiation.cc:407] Connection header received
0927 22:56:44.442055 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.442059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.442113 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.442210 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:44.443923 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.444668 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.446056 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.446452 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.446610 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.448884 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:44.448888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:44.448891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:44.448920 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:44.451198 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.451864 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.451868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.451870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.452048 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.452472 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.452477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.452480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.452876 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:44.453135 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.453561 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.453802 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:56:44.457945 24503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:56:44.464730 24503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:56:44.508430 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.477970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34364 (local address 127.8.204.131:39549)
0927 22:56:44.501519 (+ 23549us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.501525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.501549 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:56:44.501615 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.501620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.501757 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.501936 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:44.502485 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.503635 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.506331 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.507226 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.507420 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.507651 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.507913 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.508073 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23358,"thread_start_us":23176,"threads_started":1}
I20260927 22:56:44.508549 24972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.477875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39549 (local address 127.0.0.1:34364)
0927 22:56:44.478757 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.478820 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:44.479064 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:44.501928 (+ 22864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:44.501932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:44.501967 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:44.502304 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.502314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.503781 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.503786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.506168 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.506178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.507457 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.507463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.507654 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.507711 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:56:44.507868 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 22:56:44.508063 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":51,"thread_start_us":19299,"threads_started":1}
I20260927 22:56:44.509452 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.480129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.8.204.130:38431)
0927 22:56:44.495491 (+ 15362us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.495506 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.500234 (+ 4728us) server_negotiation.cc:407] Connection header received
0927 22:56:44.500375 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.500386 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.500724 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.500972 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:44.501941 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.503434 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.506301 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.507767 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.507893 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.508065 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.508951 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.509117 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15085,"thread_start_us":15364,"threads_started":1}
I20260927 22:56:44.510038 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.479815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38431 (local address 127.0.0.1:39872)
0927 22:56:44.499384 (+ 19569us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.499414 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:44.499564 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:44.501253 (+ 1689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:44.501258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:44.501281 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:44.501713 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.501726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.503575 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.503582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.506126 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.506143 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.508215 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.508221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.508403 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.508432 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:56:44.508542 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:56:44.509710 (+ 1168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19360,"mutex_wait_us":39,"thread_start_us":19215,"threads_started":1}
I20260927 22:56:44.509440 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.478636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57242 (local address 127.8.204.129:40025)
0927 22:56:44.500188 (+ 21552us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.500196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.500219 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:56:44.500307 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.500313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.500568 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.500785 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:44.501470 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.503163 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.505684 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.507165 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.507329 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.507559 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.508951 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.509118 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21384,"thread_start_us":21200,"threads_started":1}
I20260927 22:56:44.510599 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.478019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57242)
0927 22:56:44.496902 (+ 18883us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.496933 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:44.497085 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:44.500804 (+ 3719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:44.500811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:44.500839 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:44.501209 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.501221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.503343 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.503349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.505494 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.505508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.507704 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.507710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.507909 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.507939 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:56:44.508020 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 22:56:44.508156 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18717,"mutex_wait_us":103,"thread_start_us":872,"threads_started":1}
I20260927 22:56:44.512732 24894 tablet_service.cc:1467] Processing CreateTablet for tablet ac539c9f114f469ea1e470c2cded46cb (DEFAULT_TABLE table=test-table [id=d656359147d941369b1393f649df7f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:44.514300 24894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac539c9f114f469ea1e470c2cded46cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:44.513788 24761 tablet_service.cc:1467] Processing CreateTablet for tablet ac539c9f114f469ea1e470c2cded46cb (DEFAULT_TABLE table=test-table [id=d656359147d941369b1393f649df7f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:44.514149 24627 tablet_service.cc:1467] Processing CreateTablet for tablet ac539c9f114f469ea1e470c2cded46cb (DEFAULT_TABLE table=test-table [id=d656359147d941369b1393f649df7f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:56:44.515076 24761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac539c9f114f469ea1e470c2cded46cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:44.515437 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac539c9f114f469ea1e470c2cded46cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:56:44.526378 24977 tablet_bootstrap.cc:492] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Bootstrap starting.
I20260927 22:56:44.528729 24978 tablet_bootstrap.cc:492] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Bootstrap starting.
I20260927 22:56:44.530156 24977 tablet_bootstrap.cc:654] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:44.531659 24977 log.cc:824] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:44.532570 24978 tablet_bootstrap.cc:654] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:44.533962 24978 log.cc:824] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:44.534659 24979 tablet_bootstrap.cc:492] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Bootstrap starting.
I20260927 22:56:44.537245 24978 tablet_bootstrap.cc:492] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: No bootstrap required, opened a new log
I20260927 22:56:44.537540 24978 ts_tablet_manager.cc:1397] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:56:44.538197 24979 tablet_bootstrap.cc:654] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Neither blocks nor log segments found. Creating new log.
I20260927 22:56:44.539354 24979 log.cc:824] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:56:44.541078 24977 tablet_bootstrap.cc:492] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: No bootstrap required, opened a new log
I20260927 22:56:44.541337 24977 ts_tablet_manager.cc:1397] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260927 22:56:44.542135 24979 tablet_bootstrap.cc:492] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: No bootstrap required, opened a new log
I20260927 22:56:44.542371 24979 ts_tablet_manager.cc:1397] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 22:56:44.547920 24978 raft_consensus.cc:348] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.548359 24978 raft_consensus.cc:374] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:44.548448 24978 raft_consensus.cc:729] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e820f2f6cb0446785d9be5325ee63a3, State: Initialized, Role: FOLLOWER
I20260927 22:56:44.549064 24978 consensus_queue.cc:260] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.549149 24977 raft_consensus.cc:348] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.549499 24977 raft_consensus.cc:374] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:44.549610 24977 raft_consensus.cc:729] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7ba1b7db0548109c69b9ad8416cbe2, State: Initialized, Role: FOLLOWER
I20260927 22:56:44.550246 24826 heartbeater.cc:503] Master 127.8.204.190:40233 was elected leader, sending a full tablet report...
I20260927 22:56:44.550838 24978 ts_tablet_manager.cc:1428] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260927 22:56:44.551604 24977 consensus_queue.cc:260] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.552459 24979 raft_consensus.cc:348] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.552870 24979 raft_consensus.cc:374] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:56:44.553031 24979 raft_consensus.cc:729] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea7c04cfea94b71a8e499d18633a73d, State: Initialized, Role: FOLLOWER
I20260927 22:56:44.553570 24979 consensus_queue.cc:260] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.557585 24959 heartbeater.cc:503] Master 127.8.204.190:40233 was elected leader, sending a full tablet report...
I20260927 22:56:44.558564 24979 ts_tablet_manager.cc:1428] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260927 22:56:44.560027 24977 ts_tablet_manager.cc:1428] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260927 22:56:44.653728 24984 raft_consensus.cc:482] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:56:44.654065 24984 raft_consensus.cc:504] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.655746 24984 leader_election.cc:288] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549), 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
W20260927 22:56:44.663096 24693 tablet.cc:2369] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:44.663106 24960 tablet.cc:2369] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:56:44.664505 24827 tablet.cc:2369] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:56:44.665328 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.656275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:40065 (local address 127.8.204.130:38431)
0927 22:56:44.656462 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.656469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.657275 (+ 806us) server_negotiation.cc:407] Connection header received
0927 22:56:44.657449 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.657455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.657536 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.657678 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:44.658578 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.659755 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.662820 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.664136 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.664313 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.664391 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.664920 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.665054 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:56:44.665544 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.656277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38431 (local address 127.8.204.129:40065)
0927 22:56:44.657162 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.657179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:44.657298 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:44.657696 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:44.657702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:44.657765 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:44.658362 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.658382 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.659886 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.659889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.662599 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.662614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.664407 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.664414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.664642 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.664779 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 22:56:44.664861 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 22:56:44.665049 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":315,"spinlock_wait_cycles":76544,"thread_start_us":450,"threads_started":1}
I20260927 22:56:44.665565 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.656215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39549 (local address 127.8.204.129:54071)
0927 22:56:44.657097 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.657116 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:56:44.657295 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:56:44.657696 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:56:44.657702 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:56:44.657757 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:56:44.658389 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.658398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.659884 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.659889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.662601 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:56:44.662614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.664419 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.664423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:56:44.664641 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.664768 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 22:56:44.664843 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:56:44.665049 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":315,"spinlock_wait_cycles":11008,"thread_start_us":497,"threads_started":1}
I20260927 22:56:44.666028 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.656181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:54071 (local address 127.8.204.131:39549)
0927 22:56:44.656411 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.656418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.657246 (+ 828us) server_negotiation.cc:407] Connection header received
0927 22:56:44.657450 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.657455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.657536 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.657652 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:56:44.658578 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.659755 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.662813 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.664172 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.665399 (+ 1227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.665480 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.665528 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.665779 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:56:44.666780 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac539c9f114f469ea1e470c2cded46cb" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e820f2f6cb0446785d9be5325ee63a3" is_pre_election: true
I20260927 22:56:44.666949 24914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac539c9f114f469ea1e470c2cded46cb" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea7c04cfea94b71a8e499d18633a73d" is_pre_election: true
I20260927 22:56:44.667317 24781 raft_consensus.cc:2454] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 0.
I20260927 22:56:44.667515 24914 raft_consensus.cc:2454] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 0.
I20260927 22:56:44.668023 24581 leader_election.cc:302] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e820f2f6cb0446785d9be5325ee63a3, ee7ba1b7db0548109c69b9ad8416cbe2; no voters:
I20260927 22:56:44.668475 24984 raft_consensus.cc:2781] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:56:44.668629 24984 raft_consensus.cc:482] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:56:44.668728 24984 raft_consensus.cc:3037] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:44.670903 24984 raft_consensus.cc:504] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.671653 24984 leader_election.cc:288] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 election: Requested vote from peers bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549), 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
I20260927 22:56:44.672022 24914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac539c9f114f469ea1e470c2cded46cb" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea7c04cfea94b71a8e499d18633a73d"
I20260927 22:56:44.672292 24914 raft_consensus.cc:3037] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:44.672341 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac539c9f114f469ea1e470c2cded46cb" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e820f2f6cb0446785d9be5325ee63a3"
I20260927 22:56:44.672540 24781 raft_consensus.cc:3037] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:56:44.674425 24914 raft_consensus.cc:2454] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 1.
I20260927 22:56:44.674558 24781 raft_consensus.cc:2454] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 1.
I20260927 22:56:44.674835 24583 leader_election.cc:302] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bea7c04cfea94b71a8e499d18633a73d, ee7ba1b7db0548109c69b9ad8416cbe2; no voters:
I20260927 22:56:44.675186 24984 raft_consensus.cc:2781] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:56:44.675645 24984 raft_consensus.cc:686] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 LEADER]: Becoming Leader. State: Replica: ee7ba1b7db0548109c69b9ad8416cbe2, State: Running, Role: LEADER
I20260927 22:56:44.676095 24984 consensus_queue.cc:237] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:56:44.680521 24501 catalog_manager.cc:5495] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129). New cstate: current_term: 1 leader_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } health_report { overall_health: UNKNOWN } } }
I20260927 22:56:44.729930 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.724916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57258 (local address 127.8.204.129:40025)
0927 22:56:44.725097 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.725101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.725117 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:56:44.725238 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.725241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.725320 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.725443 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:56:44.725747 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.726463 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.727944 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.728325 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.728416 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.728854 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:56:44.729349 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:56:44.729458 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.729533 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.729618 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:56:44.738179 24781 raft_consensus.cc:1261] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Refusing update from remote peer ee7ba1b7db0548109c69b9ad8416cbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:44.738179 24914 raft_consensus.cc:1261] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Refusing update from remote peer ee7ba1b7db0548109c69b9ad8416cbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:56:44.739209 24988 consensus_queue.cc:1035] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:44.739604 24984 consensus_queue.cc:1035] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:56:44.751071 24991 mvcc.cc:205] Tried to move back new op lower bound from 7334092000197144576 to 7334091999957098496. Current Snapshot: MvccSnapshot[applied={T|T < 7334092000197144576}]
I20260927 22:56:44.764746 24503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:44.765044 24503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:56:44.768541 24503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Sending DeleteTablet for 3 replicas of tablet ac539c9f114f469ea1e470c2cded46cb
I20260927 22:56:44.769579 24894 tablet_service.cc:1514] Processing DeleteTablet for tablet ac539c9f114f469ea1e470c2cded46cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:44 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34364
I20260927 22:56:44.769713 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet ac539c9f114f469ea1e470c2cded46cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:44 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:57242
I20260927 22:56:44.769685 24761 tablet_service.cc:1514] Processing DeleteTablet for tablet ac539c9f114f469ea1e470c2cded46cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:56:44 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:39872
I20260927 22:56:44.770835 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 24564
I20260927 22:56:44.782014 25003 tablet_replica.cc:330] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: stopping tablet replica
I20260927 22:56:44.782606 25002 tablet_replica.cc:330] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: stopping tablet replica
I20260927 22:56:44.782941 25001 tablet_replica.cc:330] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: stopping tablet replica
I20260927 22:56:44.783491 25001 raft_consensus.cc:2229] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:44.784004 25003 raft_consensus.cc:2229] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:56:44.784065 25001 raft_consensus.cc:2258] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:44.783572 25002 raft_consensus.cc:2229] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:56:44.786119 25001 ts_tablet_manager.cc:1905] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:44.786764 25002 raft_consensus.cc:2258] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:44.785228 25003 raft_consensus.cc:2258] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:56:44.788972 25002 ts_tablet_manager.cc:1905] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:44.793484 25001 ts_tablet_manager.cc:1918] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:44.793635 25001 log.cc:1196] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/ac539c9f114f469ea1e470c2cded46cb
Sep 27 22:56:44 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:56:44.794147 25001 ts_tablet_manager.cc:1939] T ac539c9f114f469ea1e470c2cded46cb P bea7c04cfea94b71a8e499d18633a73d: Deleting consensus metadata
I20260927 22:56:44.795687 25002 ts_tablet_manager.cc:1918] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:44.796154 24490 catalog_manager.cc:4841] TS bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549): tablet ac539c9f114f469ea1e470c2cded46cb (table test-table [id=d656359147d941369b1393f649df7f34]) successfully deleted
I20260927 22:56:44.798491 25003 ts_tablet_manager.cc:1905] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:56:44.804008 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.772140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.8.204.129:40025)
0927 22:56:44.772341 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.772350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.782268 (+ 9918us) server_negotiation.cc:407] Connection header received
0927 22:56:44.782519 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.782528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.782627 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.782747 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:44.784956 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.786324 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.788741 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.789392 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.789484 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.795530 (+ 6046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:44.795545 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:44.795563 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:44.795616 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:44.799653 (+ 4037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.800657 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.800670 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.800674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.800860 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.801330 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.801336 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.801339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.801748 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:44.802072 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.803365 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.803616 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:56:44.804697 24687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:44.806077 25002 log.cc:1196] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/ac539c9f114f469ea1e470c2cded46cb
I20260927 22:56:44.806641 25002 ts_tablet_manager.cc:1939] T ac539c9f114f469ea1e470c2cded46cb P 2e820f2f6cb0446785d9be5325ee63a3: Deleting consensus metadata
I20260927 22:56:44.808260 24488 catalog_manager.cc:4841] TS 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431): tablet ac539c9f114f469ea1e470c2cded46cb (table test-table [id=d656359147d941369b1393f649df7f34]) successfully deleted
I20260927 22:56:44.905370 25003 ts_tablet_manager.cc:1918] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:56:44.905690 25003 log.cc:1196] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/ac539c9f114f469ea1e470c2cded46cb
I20260927 22:56:44.906224 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 24564
W20260927 22:56:44.922609 24488 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.129:40025 (error 108)
I20260927 22:56:44.923048 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 24698
I20260927 22:56:44.924974 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.924466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57274)
0927 22:56:44.924700 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.924820 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260927 22:56:44.925200 24488 catalog_manager.cc:4584] TS ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129:40025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac539c9f114f469ea1e470c2cded46cb: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Sep 27 22:56:44 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:56:44.938817 24976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.923824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.8.204.130:38431)
0927 22:56:44.924001 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:44.924007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:44.924137 (+ 130us) server_negotiation.cc:407] Connection header received
0927 22:56:44.924361 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:44.924366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:44.924427 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:44.924530 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:44.925981 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.926858 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.928283 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:44.928689 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:44.928819 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:44.933068 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:44.933079 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:44.933092 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:44.933121 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:44.936041 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.936706 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.936721 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.936724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.936843 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:44.937204 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:44.937210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:44.937213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:44.937580 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:44.937895 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:44.938225 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 22:56:44.938488 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:56:44.939497 24821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:44.979010 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.978454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57278)
0927 22:56:44.978683 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.978818 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:56:44.980082 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:44.979686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57290)
0927 22:56:44.979856 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:44.979938 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:56:45.029910 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 24698
I20260927 22:56:45.040488 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:45.040048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57294)
0927 22:56:45.040247 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:45.040334 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 22:56:45.041461 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:45.041134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57306)
0927 22:56:45.041269 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:45.041345 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 22:56:45.047737 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 24831
Sep 27 22:56:45 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:56:45.062706 24973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:45.048438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34378 (local address 127.8.204.131:39549)
0927 22:56:45.048614 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:56:45.048618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:56:45.048637 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:56:45.048757 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:56:45.048768 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:56:45.048841 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:56:45.048923 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:56:45.050169 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:45.050887 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:45.052480 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:56:45.052965 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:56:45.053058 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:56:45.057524 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:56:45.057538 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:56:45.057553 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:56:45.057585 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:56:45.060409 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:45.061066 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:45.061073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:45.061075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:45.061202 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:56:45.061502 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:56:45.061505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:56:45.061507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:56:45.061845 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:56:45.062052 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:56:45.062293 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:56:45.062456 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:56:45.063393 24954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:45.121973 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:45.121474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57318)
0927 22:56:45.121724 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:45.121810 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 22:56:45.123032 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:56:45.122644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:57332)
0927 22:56:45.122860 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:56:45.122927 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 22:56:45.151831 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 24831
I20260927 22:56:45.168705 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 24471
I20260927 22:56:45.169821 24533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:56:45.282725 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 24471
I20260927 22:57:05.304052 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40233
--webserver_interface=127.8.204.190
--webserver_port=42221
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:05.628484 25010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:05.628736 25010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:05.628823 25010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:05.635828 25010 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:57:05.635918 25010 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:57:05.635985 25010 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:57:05.636075 25010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:57:05.636169 25010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:05.636281 25010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:05.636363 25010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:57:05.636446 25010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:57:05.646193 25010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:05.647505 25010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:05.648809 25010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:05.671411 25017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:05.671394 25019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:05.672340 25016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:05.672350 25010 server_base.cc:1034] running on GCE node
I20260927 22:57:05.673422 25010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:05.675081 25010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:05.676329 25010 hybrid_clock.cc:648] HybridClock initialized: now 1790549825676265 us; error 37 us; skew 500 ppm
I20260927 22:57:05.676785 25010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:05 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:05.680510 25010 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:57:05.682190 25010 webserver.cc:460] Webserver started at http://127.8.204.190:42221/ using document root <none> and password file <none>
I20260927 22:57:05.683041 25010 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:05.683200 25010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:05.689141 25010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:57:05.692076 25026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:05.693385 25010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:05.693586 25010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "4e8d418a13ba4357a6fecc6e59cd34d2"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-40vx"
server_key: "aa7da9650d2b617641b8720f5d165917"
server_key_iv: "fc983e749df514e9c2995a32358a2a4b"
server_key_version: "encryptionkey@0"
I20260927 22:57:05.694598 25010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:05.707914 25010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:05.711946 25010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:05.712229 25010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:05.730854 25010 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40233
I20260927 22:57:05.730878 25077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40233 every 8 connection(s)
I20260927 22:57:05.732478 25010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:57:05.737150 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25010
I20260927 22:57:05.738210 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:40025
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:40233
--webserver_port=37797
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:57:05.740688 25078 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:57:05.746145 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap starting.
I20260927 22:57:05.766206 25078 log.cc:824] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:05.778537 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:57:05.779104 25078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap complete.
I20260927 22:57:05.788373 25078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:05.789199 25078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8d418a13ba4357a6fecc6e59cd34d2, State: Initialized, Role: FOLLOWER
I20260927 22:57:05.789918 25078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:05.790158 25078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:57:05.790277 25078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:57:05.790462 25078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:57:05.794302 25078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:05.794798 25078 leader_election.cc:302] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e8d418a13ba4357a6fecc6e59cd34d2; no voters:
I20260927 22:57:05.795519 25078 leader_election.cc:288] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:57:05.795646 25082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:57:05.797050 25082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 LEADER]: Becoming Leader. State: Replica: 4e8d418a13ba4357a6fecc6e59cd34d2, State: Running, Role: LEADER
I20260927 22:57:05.797721 25082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:05.797946 25078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:57:05.800736 25084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e8d418a13ba4357a6fecc6e59cd34d2. Latest consensus state: current_term: 2 leader_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } } }
I20260927 22:57:05.801147 25084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:05.801725 25083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } } }
I20260927 22:57:05.802044 25083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:05.820330 25089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:57:05.829767 25089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:57:05.831561 25089 catalog_manager.cc:1259] Loaded cluster ID: fe8a013763e94d51baf95747422a06b7
I20260927 22:57:05.831791 25089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:57:05.837531 25089 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:57:05.843611 25089 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Loaded TSK: 0
I20260927 22:57:05.882160 25089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Generated new TSK 1
I20260927 22:57:05.882583 25089 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:57:06.125485 25080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:06.125793 25080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:06.125965 25080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:06.133172 25080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:06.133332 25080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:06.133544 25080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:57:06.143360 25080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:40025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=37797
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:06.144697 25080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:06.146032 25080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:06.160923 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:06.161082 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:06.160894 25108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:06.161415 25080 server_base.cc:1034] running on GCE node
I20260927 22:57:06.162209 25080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:06.163237 25080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:06.164481 25080 hybrid_clock.cc:648] HybridClock initialized: now 1790549826164430 us; error 48 us; skew 500 ppm
I20260927 22:57:06.164795 25080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:06 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:06.168011 25080 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:57:06.169641 25080 webserver.cc:460] Webserver started at http://127.8.204.129:37797/ using document root <none> and password file <none>
I20260927 22:57:06.170342 25080 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:06.170488 25080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:06.176155 25080 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:57:06.178992 25115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:06.180434 25080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:57:06.180586 25080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "ee7ba1b7db0548109c69b9ad8416cbe2"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "47bc583ed2fb1b336e3263c7137fdd41"
server_key_iv: "e4c71591938d564957bdd1bcd8f438c3"
server_key_version: "encryptionkey@0"
I20260927 22:57:06.181536 25080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:06.205377 25080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:06.209643 25080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:06.209923 25080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:06.210901 25080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:06.212685 25122 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 22:57:06.217998 25122 ts_tablet_manager.cc:1852] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 22:57:06.218977 25122 ts_tablet_manager.cc:1905] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:06.226128 25122 ts_tablet_manager.cc:1918] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:06.226408 25122 ts_tablet_manager.cc:1939] T ac539c9f114f469ea1e470c2cded46cb P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting consensus metadata
I20260927 22:57:06.226816 25080 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 22:57:06.226917 25080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260927 22:57:06.227065 25080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:06.227156 25080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:06.259817 25080 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:40025
I20260927 22:57:06.259864 25228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:40025 every 8 connection(s)
I20260927 22:57:06.261068 25080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:57:06.266044 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25080
I20260927 22:57:06.266654 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:38431
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:40233
--webserver_port=41895
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:57:06 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:06.287037 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:06.263923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:41175 (local address 127.8.204.190:40233)
0927 22:57:06.264445 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:06.264456 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:06.265147 (+ 691us) server_negotiation.cc:407] Connection header received
0927 22:57:06.267631 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:06.267651 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:06.268123 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:06.268520 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:06.269985 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.272026 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.273620 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.274363 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.274530 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:06.278987 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:06.278994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:06.279007 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:06.279042 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:06.282034 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:06.282990 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:06.282998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:06.283006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:06.283266 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:06.283788 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:06.283791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:06.283793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:06.284363 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:06.284544 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:06.285642 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0927 22:57:06.286032 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260927 22:57:06.287494 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:06.264252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:41175)
0927 22:57:06.264972 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:06.265015 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:06.266987 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:06.268656 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:06.268663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:06.269219 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:06.269770 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:06.269794 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.272198 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.272202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:06.273462 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:06.273474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.274628 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.274637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:06.274955 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:06.275671 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:06.275704 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:06.278709 (+ 3005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:06.282175 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:06.282185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:06.282204 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:06.282824 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:06.283440 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:06.283448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:06.283454 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:06.283679 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:06.284534 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:06.284545 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:06.285009 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 22:57:06.285360 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 22:57:06.285976 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":288,"threads_started":1}
I20260927 22:57:06.290338 25229 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40233
I20260927 22:57:06.290663 25229 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:06.291421 25229 heartbeater.cc:511] Master 127.8.204.190:40233 requested a full tablet report, sending...
I20260927 22:57:06.293584 25043 ts_manager.cc:194] Registered new tserver with Master: ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129:40025)
I20260927 22:57:06.295044 25043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:41175
W20260927 22:57:06.593909 25233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:06.594241 25233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:06.594424 25233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:06.601969 25233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:06.602114 25233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:06.602304 25233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:57:06.612108 25233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:38431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=41895
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:06.613392 25233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:06.614673 25233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:06.627028 25238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:06.627106 25239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:06.627130 25241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:06.628123 25233 server_base.cc:1034] running on GCE node
I20260927 22:57:06.628719 25233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:06.629578 25233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:06.630800 25233 hybrid_clock.cc:648] HybridClock initialized: now 1790549826630759 us; error 54 us; skew 500 ppm
I20260927 22:57:06.631161 25233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:06 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:06.634109 25233 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:57:06.635504 25233 webserver.cc:460] Webserver started at http://127.8.204.130:41895/ using document root <none> and password file <none>
I20260927 22:57:06.636179 25233 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:06.636332 25233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:06.641474 25233 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260927 22:57:06.643981 25248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:06.645210 25233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:57:06.645371 25233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "2e820f2f6cb0446785d9be5325ee63a3"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "df0d6584e054d3b3833b34de46814b5d"
server_key_iv: "16b114eb990434f51c2c8a54fac188be"
server_key_version: "encryptionkey@0"
I20260927 22:57:06.646282 25233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:06.660264 25233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:06.664868 25233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:06.665141 25233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:06.666090 25233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:06.667639 25233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:06.667724 25233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:06.667877 25233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:06.667966 25233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:06.700263 25233 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38431
I20260927 22:57:06.700314 25360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38431 every 8 connection(s)
I20260927 22:57:06.701725 25233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:57:06.702594 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25233
I20260927 22:57:06.703416 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:39549
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:40233
--webserver_port=44501
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:57:06 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:06.722043 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:06.707480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:32971 (local address 127.8.204.190:40233)
0927 22:57:06.707690 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:06.707695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:06.707720 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:57:06.707794 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:06.707798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:06.707891 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:06.708007 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:06.709542 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.710286 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.712122 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.712531 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.712668 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:06.715981 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:06.715986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:06.715989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:06.716022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:06.718490 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:06.719329 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:06.719338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:06.719344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:06.719549 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:06.719970 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:06.719978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:06.719984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:06.720459 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:06.720631 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:06.721278 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 22:57:06.721572 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:57:06.722589 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:06.704908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:32971)
0927 22:57:06.705727 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:06.705778 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:06.707514 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:06.708242 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:06.708251 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:06.708770 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:06.709315 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:06.709360 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.710455 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.710462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:06.711966 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:06.711978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:06.712748 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:06.712757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:06.712978 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:06.713632 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:06.713654 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:06.715746 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:06.718635 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:06.718643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:06.718659 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:06.719114 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:06.719665 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:06.719669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:06.719671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:06.719839 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:06.720577 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:06.720582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:06.720880 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 22:57:06.721107 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 22:57:06.721494 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":332,"threads_started":1}
I20260927 22:57:06.724138 25361 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40233
I20260927 22:57:06.724450 25361 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:06.725144 25361 heartbeater.cc:511] Master 127.8.204.190:40233 requested a full tablet report, sending...
I20260927 22:57:06.726588 25043 ts_manager.cc:194] Registered new tserver with Master: 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
I20260927 22:57:06.727492 25043 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:32971
W20260927 22:57:07.038684 25364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:07.038969 25364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:07.039065 25364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:07.046418 25364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:07.046528 25364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:07.046640 25364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:57:07.056463 25364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:39549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=44501
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:07.057744 25364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:07.059028 25364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:07.072032 25369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:07.072057 25370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:07.072175 25372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:07.072722 25364 server_base.cc:1034] running on GCE node
I20260927 22:57:07.073323 25364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:07.074259 25364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:07.075518 25364 hybrid_clock.cc:648] HybridClock initialized: now 1790549827075433 us; error 44 us; skew 500 ppm
I20260927 22:57:07.075836 25364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:07 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549827, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:07.082587 25364 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:57:07.084124 25364 webserver.cc:460] Webserver started at http://127.8.204.131:44501/ using document root <none> and password file <none>
I20260927 22:57:07.084910 25364 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:07.085071 25364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:07.091001 25364 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:57:07.093798 25379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:07.095165 25364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:57:07.095368 25364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "bea7c04cfea94b71a8e499d18633a73d"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-40vx"
server_key: "99923efcc04589b31e68157f4c138e1a"
server_key_iv: "728f0dc0f02be10c3bbde35aa8bd96b6"
server_key_version: "encryptionkey@0"
I20260927 22:57:07.096346 25364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:07.119642 25364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:07.123410 25364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:07.123687 25364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:07.124581 25364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:07.125826 25364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:07.125954 25364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:07.126080 25364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:07.126186 25364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:07.158430 25364 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:39549
I20260927 22:57:07.158517 25491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:39549 every 8 connection(s)
I20260927 22:57:07.160046 25364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:57:07.166649 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25364
Sep 27 22:57:07 dist-test-slave-40vx krb5kdc[24458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549827, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:07.182396 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.162413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:37741 (local address 127.8.204.190:40233)
0927 22:57:07.162599 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.162603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.163808 (+ 1205us) server_negotiation.cc:407] Connection header received
0927 22:57:07.165604 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.165613 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.165704 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.165812 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:07.167797 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.168906 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.170509 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.170964 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.171060 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.175417 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:07.175430 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:07.175433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:07.175462 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:07.179197 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.179931 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.179936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.179939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.180072 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.180540 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.180543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.180546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.180885 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:07.181005 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.181785 (+ 780us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.181993 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"spinlock_wait_cycles":6528}
I20260927 22:57:07.182688 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.167717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.8.204.190:40233)
0927 22:57:07.168510 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.168515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.168530 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:57:07.168568 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.168572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.168624 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.168711 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:07.170314 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.171353 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.172863 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.173545 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.173671 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.176639 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:07.176646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:07.176651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:07.176699 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:07.179960 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.180706 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.180709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.180712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.180820 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.181233 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.181236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.181238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.181466 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:07.181672 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.182015 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.182207 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":565,"threads_started":1}
I20260927 22:57:07.183485 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.162891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:37741)
0927 22:57:07.163637 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.163677 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.165342 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.166046 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.166054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.166802 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:07.167634 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.167655 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.169076 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.169080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.170351 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.170366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.171170 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.171177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.171494 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.172225 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:07.172247 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:07.175191 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:07.179372 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:07.179378 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:07.179390 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:07.179802 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:07.180185 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:07.180190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:07.180193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:07.180427 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:07.181013 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:07.181022 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:07.181402 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.181686 (+ 284us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.182203 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260927 22:57:07.185905 25492 heartbeater.cc:347] Connected to a master server at 127.8.204.190:40233
I20260927 22:57:07.186244 25492 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:07.186887 25492 heartbeater.cc:511] Master 127.8.204.190:40233 requested a full tablet report, sending...
I20260927 22:57:07.188329 25042 ts_manager.cc:194] Registered new tserver with Master: bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549)
I20260927 22:57:07.189133 25042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:37741
I20260927 22:57:07.197644 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:57:07.204077 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.198597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40516 (local address 127.8.204.190:40233)
0927 22:57:07.198775 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.198784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.199321 (+ 537us) server_negotiation.cc:407] Connection header received
0927 22:57:07.199647 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.199651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.199791 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.199914 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:57:07.200342 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.201033 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.202431 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.202828 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.202917 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.203406 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:57:07.203620 (+ 214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:57:07.203880 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40516: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 22:57:07.204171 25496 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40516: token expired
W20260927 22:57:07.204771 24964 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.204.190:40233)
I20260927 22:57:07.216892 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.205367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40528 (local address 127.8.204.190:40233)
0927 22:57:07.205550 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.205554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.205569 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:57:07.205604 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.205607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.205655 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.205734 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:07.207015 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.207834 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.209206 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.209569 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.209651 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.212117 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:07.212122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:07.212125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:07.212154 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:07.214588 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.215198 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.215202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.215204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.215416 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.215751 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.215754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.215756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.216047 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:07.216243 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.216481 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.216638 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:57:07.220145 25042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:57:07.227363 25042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:57:07.252738 25502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.239529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39549 (local address 127.0.0.1:45884)
0927 22:57:07.240564 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.240605 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.240771 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.242122 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.242133 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.242178 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.242732 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.242748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.245725 (+ 2977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.245734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.248957 (+ 3223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.248978 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.251566 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.251573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.251876 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.251956 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.252101 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.252354 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":842,"threads_started":1}
I20260927 22:57:07.254657 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.239695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.8.204.131:39549)
0927 22:57:07.241428 (+ 1733us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.241437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.241470 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:57:07.241571 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.241582 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.241868 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.243009 (+ 1141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.243387 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.245517 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.249252 (+ 3735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.251242 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.253621 (+ 2379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.253969 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.254058 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.254282 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562,"thread_start_us":1421,"threads_started":1}
I20260927 22:57:07.258711 25427 tablet_service.cc:1467] Processing CreateTablet for tablet d403d8c1f0764b9c948d02149f69f36c (DEFAULT_TABLE table=test-table [id=ea99311a8c304bada6f477c5f31270c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:07.260280 25427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d403d8c1f0764b9c948d02149f69f36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:07.261722 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.241587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38431 (local address 127.0.0.1:36140)
0927 22:57:07.253245 (+ 11658us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.253263 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.253378 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.255377 (+ 1999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.255381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.255407 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.255742 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.255755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.257394 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.257399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.259090 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.259104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.260940 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.260950 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.261128 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.261160 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.261227 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.261354 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11511,"thread_start_us":11370,"threads_started":1}
I20260927 22:57:07.262399 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.239938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.0.0.1:51414)
0927 22:57:07.241356 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.241372 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.241482 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.254017 (+ 12535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.254022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.254048 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.254400 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.254427 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.256208 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.256214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.258302 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.258317 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.260484 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.260491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.261734 (+ 1243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.261775 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.261970 (+ 195us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.262098 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":1135,"threads_started":1}
I20260927 22:57:07.262719 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.241901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.8.204.130:38431)
0927 22:57:07.254649 (+ 12748us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.254658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.254690 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:57:07.254783 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.254792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.255065 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.255252 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.255917 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.257256 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.259292 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.260689 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.261989 (+ 1300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.262176 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.262245 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.262408 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12589,"thread_start_us":12494,"threads_started":1}
I20260927 22:57:07.263182 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.240570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.8.204.129:40025)
0927 22:57:07.252763 (+ 12193us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.252773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.252805 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:57:07.252920 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.252928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.253203 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.254051 (+ 848us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.254608 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.256044 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.258484 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.259996 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.260128 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.260322 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.262661 (+ 2339us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.262846 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12019,"thread_start_us":11905,"threads_started":1}
I20260927 22:57:07.264884 25164 tablet_service.cc:1467] Processing CreateTablet for tablet d403d8c1f0764b9c948d02149f69f36c (DEFAULT_TABLE table=test-table [id=ea99311a8c304bada6f477c5f31270c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:07.266314 25164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d403d8c1f0764b9c948d02149f69f36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:07.266556 25296 tablet_service.cc:1467] Processing CreateTablet for tablet d403d8c1f0764b9c948d02149f69f36c (DEFAULT_TABLE table=test-table [id=ea99311a8c304bada6f477c5f31270c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:07.268044 25296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d403d8c1f0764b9c948d02149f69f36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:07.274375 25508 tablet_bootstrap.cc:492] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Bootstrap starting.
I20260927 22:57:07.277520 25508 tablet_bootstrap.cc:654] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:07.278548 25508 log.cc:824] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:07.280540 25510 tablet_bootstrap.cc:492] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Bootstrap starting.
I20260927 22:57:07.281911 25508 tablet_bootstrap.cc:492] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: No bootstrap required, opened a new log
I20260927 22:57:07.282181 25508 ts_tablet_manager.cc:1397] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 22:57:07.283028 25511 tablet_bootstrap.cc:492] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Bootstrap starting.
I20260927 22:57:07.284065 25510 tablet_bootstrap.cc:654] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:07.285251 25510 log.cc:824] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:07.286166 25511 tablet_bootstrap.cc:654] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:07.287204 25511 log.cc:824] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:07.288622 25510 tablet_bootstrap.cc:492] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: No bootstrap required, opened a new log
I20260927 22:57:07.288872 25510 ts_tablet_manager.cc:1397] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:57:07.292609 25508 raft_consensus.cc:348] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.293048 25508 raft_consensus.cc:374] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:07.293151 25508 raft_consensus.cc:729] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7ba1b7db0548109c69b9ad8416cbe2, State: Initialized, Role: FOLLOWER
I20260927 22:57:07.293596 25511 tablet_bootstrap.cc:492] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: No bootstrap required, opened a new log
I20260927 22:57:07.293845 25511 ts_tablet_manager.cc:1397] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260927 22:57:07.293758 25508 consensus_queue.cc:260] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.294804 25229 heartbeater.cc:503] Master 127.8.204.190:40233 was elected leader, sending a full tablet report...
I20260927 22:57:07.295440 25508 ts_tablet_manager.cc:1428] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260927 22:57:07.298956 25510 raft_consensus.cc:348] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.299464 25510 raft_consensus.cc:374] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:07.299568 25510 raft_consensus.cc:729] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bea7c04cfea94b71a8e499d18633a73d, State: Initialized, Role: FOLLOWER
I20260927 22:57:07.300204 25510 consensus_queue.cc:260] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.301316 25492 heartbeater.cc:503] Master 127.8.204.190:40233 was elected leader, sending a full tablet report...
I20260927 22:57:07.302004 25510 ts_tablet_manager.cc:1428] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 22:57:07.304442 25511 raft_consensus.cc:348] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.304903 25511 raft_consensus.cc:374] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:07.305006 25511 raft_consensus.cc:729] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e820f2f6cb0446785d9be5325ee63a3, State: Initialized, Role: FOLLOWER
I20260927 22:57:07.305657 25511 consensus_queue.cc:260] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.306572 25361 heartbeater.cc:503] Master 127.8.204.190:40233 was elected leader, sending a full tablet report...
I20260927 22:57:07.307197 25511 ts_tablet_manager.cc:1428] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
W20260927 22:57:07.411686 25493 tablet.cc:2369] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:57:07.453792 25362 tablet.cc:2369] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:57:07.513510 25230 tablet.cc:2369] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:57:07.540701 25514 raft_consensus.cc:482] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:57:07.541095 25514 raft_consensus.cc:504] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.543200 25514 leader_election.cc:288] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549), 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
I20260927 22:57:07.547928 25515 raft_consensus.cc:482] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:57:07.548213 25515 raft_consensus.cc:504] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.549983 25515 leader_election.cc:288] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129:40025), 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
I20260927 22:57:07.552578 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.543738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:42337 (local address 127.8.204.130:38431)
0927 22:57:07.543967 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.543976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.544663 (+ 687us) server_negotiation.cc:407] Connection header received
0927 22:57:07.544765 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.544770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.544837 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.544938 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.545663 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.546728 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.549966 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.551357 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.551486 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.551562 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.552167 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.552320 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:57:07.552799 25518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.543811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38431 (local address 127.8.204.129:42337)
0927 22:57:07.544543 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.544557 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.544663 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.544969 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.544974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.545003 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.545469 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.545488 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.546885 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.546890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.549779 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.549793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.551748 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.551754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.551966 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.552013 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.552123 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.552267 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":194,"thread_start_us":483,"threads_started":1}
I20260927 22:57:07.553354 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.543494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:39549 (local address 127.8.204.129:56573)
0927 22:57:07.544394 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.544421 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.544567 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.544968 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.544974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.545065 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.545543 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.545556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.546885 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.546890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.549778 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.549793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.552457 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.552464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.552638 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.552688 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.552776 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.552922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":260,"spinlock_wait_cycles":24960,"thread_start_us":437,"threads_started":1}
I20260927 22:57:07.553894 25316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403d8c1f0764b9c948d02149f69f36c" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e820f2f6cb0446785d9be5325ee63a3" is_pre_election: true
I20260927 22:57:07.554472 25316 raft_consensus.cc:2454] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 0.
I20260927 22:57:07.555709 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.544294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:56573 (local address 127.8.204.131:39549)
0927 22:57:07.544492 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.544500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.544522 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:57:07.544736 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.544742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.544814 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.544913 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.545677 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.546734 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.549983 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.551355 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.551486 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.551581 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.555291 (+ 3710us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.555434 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":77}
I20260927 22:57:07.555876 25117 leader_election.cc:302] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e820f2f6cb0446785d9be5325ee63a3, ee7ba1b7db0548109c69b9ad8416cbe2; no voters:
I20260927 22:57:07.556708 25447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403d8c1f0764b9c948d02149f69f36c" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea7c04cfea94b71a8e499d18633a73d" is_pre_election: true
I20260927 22:57:07.557149 25447 raft_consensus.cc:2454] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 0.
I20260927 22:57:07.557614 25514 raft_consensus.cc:2781] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:57:07.557770 25514 raft_consensus.cc:482] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:57:07.557898 25514 raft_consensus.cc:3037] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:07.559158 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.550364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:40025 (local address 127.8.204.131:47597)
0927 22:57:07.550696 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.550712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.550811 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.551367 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.551376 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.551411 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.552245 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.552257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.553733 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.553739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.556835 (+ 3096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.556866 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.558434 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.558440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.558634 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.558656 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.558758 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.558892 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":516}
I20260927 22:57:07.559158 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.550342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:47597 (local address 127.8.204.129:40025)
0927 22:57:07.550520 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.550528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.550933 (+ 405us) server_negotiation.cc:407] Connection header received
0927 22:57:07.551000 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.551008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.551089 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.551180 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.552552 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.553571 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.557005 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.558267 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.558438 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.558507 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.558790 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.558915 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:57:07.559959 25184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403d8c1f0764b9c948d02149f69f36c" candidate_uuid: "bea7c04cfea94b71a8e499d18633a73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" is_pre_election: true
I20260927 22:57:07.560465 25514 raft_consensus.cc:504] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.560801 25184 raft_consensus.cc:2379] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bea7c04cfea94b71a8e499d18633a73d in current term 1: Already voted for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in this term.
I20260927 22:57:07.561012 25514 leader_election.cc:288] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 election: Requested vote from peers bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549), 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431)
I20260927 22:57:07.561379 25447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403d8c1f0764b9c948d02149f69f36c" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bea7c04cfea94b71a8e499d18633a73d"
I20260927 22:57:07.561574 25447 raft_consensus.cc:3037] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:07.561841 25316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403d8c1f0764b9c948d02149f69f36c" candidate_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e820f2f6cb0446785d9be5325ee63a3"
I20260927 22:57:07.562036 25316 raft_consensus.cc:3037] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:07.564069 25447 raft_consensus.cc:2454] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 1.
I20260927 22:57:07.564432 25119 leader_election.cc:302] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bea7c04cfea94b71a8e499d18633a73d, ee7ba1b7db0548109c69b9ad8416cbe2; no voters:
I20260927 22:57:07.564726 25514 raft_consensus.cc:2781] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:07.565251 25514 raft_consensus.cc:686] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 LEADER]: Becoming Leader. State: Replica: ee7ba1b7db0548109c69b9ad8416cbe2, State: Running, Role: LEADER
I20260927 22:57:07.565258 25316 raft_consensus.cc:2454] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in term 1.
I20260927 22:57:07.565773 25514 consensus_queue.cc:237] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } }
I20260927 22:57:07.566849 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.551822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:46503 (local address 127.8.204.130:38431)
0927 22:57:07.553641 (+ 1819us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.553648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.553671 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:57:07.553735 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.553742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.553818 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.553907 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:07.555017 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.562115 (+ 7098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.564684 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.565936 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.566117 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.566187 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.566460 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.566579 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1554,"thread_start_us":3283,"threads_started":1}
I20260927 22:57:07.566872 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.550770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38431 (local address 127.8.204.131:46503)
0927 22:57:07.552022 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:07.552034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:07.552151 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:07.554055 (+ 1904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:07.554060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:07.554081 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:07.554513 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.554526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.562291 (+ 7765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.562298 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.564476 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:07.564490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.566139 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.566145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:07.566299 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.566322 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:57:07.566437 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 22:57:07.566569 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":990,"threads_started":1}
I20260927 22:57:07.567478 25316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d403d8c1f0764b9c948d02149f69f36c" candidate_uuid: "bea7c04cfea94b71a8e499d18633a73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e820f2f6cb0446785d9be5325ee63a3" is_pre_election: true
I20260927 22:57:07.567746 25316 raft_consensus.cc:2379] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bea7c04cfea94b71a8e499d18633a73d in current term 1: Already voted for candidate ee7ba1b7db0548109c69b9ad8416cbe2 in this term.
I20260927 22:57:07.568351 25381 leader_election.cc:302] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bea7c04cfea94b71a8e499d18633a73d; no voters: 2e820f2f6cb0446785d9be5325ee63a3, ee7ba1b7db0548109c69b9ad8416cbe2
I20260927 22:57:07.568795 25515 raft_consensus.cc:2726] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 22:57:07.570737 25042 catalog_manager.cc:5495] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129). New cstate: current_term: 1 leader_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee7ba1b7db0548109c69b9ad8416cbe2" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 40025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 } health_report { overall_health: UNKNOWN } } }
I20260927 22:57:07.625257 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.620210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.8.204.129:40025)
0927 22:57:07.620420 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.620427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.620481 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:57:07.620690 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.620694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.620772 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.620899 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:57:07.621229 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.621987 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.623368 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.623787 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.623892 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.624325 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:57:07.624802 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:57:07.624908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.624976 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.625039 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 22:57:07.632766 25316 raft_consensus.cc:1261] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Refusing update from remote peer ee7ba1b7db0548109c69b9ad8416cbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:07.632819 25447 raft_consensus.cc:1261] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Refusing update from remote peer ee7ba1b7db0548109c69b9ad8416cbe2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:07.633670 25521 consensus_queue.cc:1035] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e820f2f6cb0446785d9be5325ee63a3" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:07.633998 25514 consensus_queue.cc:1035] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bea7c04cfea94b71a8e499d18633a73d" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 39549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:07.644138 25523 mvcc.cc:205] Tried to move back new op lower bound from 7334092093975076864 to 7334092093713227776. Current Snapshot: MvccSnapshot[applied={T|T < 7334092093975076864}]
I20260927 22:57:07.659756 25042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:07.660032 25042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:07.663607 25042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Sending DeleteTablet for 3 replicas of tablet d403d8c1f0764b9c948d02149f69f36c
I20260927 22:57:07.664687 25427 tablet_service.cc:1514] Processing DeleteTablet for tablet d403d8c1f0764b9c948d02149f69f36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:07 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:45884
I20260927 22:57:07.665109 25164 tablet_service.cc:1514] Processing DeleteTablet for tablet d403d8c1f0764b9c948d02149f69f36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:07 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:51414
I20260927 22:57:07.665405 25296 tablet_service.cc:1514] Processing DeleteTablet for tablet d403d8c1f0764b9c948d02149f69f36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:07 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:36140
I20260927 22:57:07.665766 25532 tablet_replica.cc:330] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: stopping tablet replica
I20260927 22:57:07.666205 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25080
I20260927 22:57:07.666343 25532 raft_consensus.cc:2229] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:07.678285 25534 tablet_replica.cc:330] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: stopping tablet replica
I20260927 22:57:07.678752 25534 raft_consensus.cc:2229] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:07.679390 25534 raft_consensus.cc:2258] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:07.680450 25533 tablet_replica.cc:330] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: stopping tablet replica
I20260927 22:57:07.680876 25533 raft_consensus.cc:2229] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:57:07.666957 25532 raft_consensus.cc:2258] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:07.681665 25533 raft_consensus.cc:2258] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:07.681711 25534 ts_tablet_manager.cc:1905] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:07.683818 25533 ts_tablet_manager.cc:1905] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:07.689338 25534 ts_tablet_manager.cc:1918] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:07.689499 25534 log.cc:1196] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/d403d8c1f0764b9c948d02149f69f36c
I20260927 22:57:07.690032 25534 ts_tablet_manager.cc:1939] T d403d8c1f0764b9c948d02149f69f36c P 2e820f2f6cb0446785d9be5325ee63a3: Deleting consensus metadata
I20260927 22:57:07.691138 25532 ts_tablet_manager.cc:1905] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:07.691354 25533 ts_tablet_manager.cc:1918] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:07.691510 25533 log.cc:1196] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/d403d8c1f0764b9c948d02149f69f36c
I20260927 22:57:07.692062 25533 ts_tablet_manager.cc:1939] T d403d8c1f0764b9c948d02149f69f36c P ee7ba1b7db0548109c69b9ad8416cbe2: Deleting consensus metadata
I20260927 22:57:07.691954 25028 catalog_manager.cc:4841] TS 2e820f2f6cb0446785d9be5325ee63a3 (127.8.204.130:38431): tablet d403d8c1f0764b9c948d02149f69f36c (table test-table [id=ea99311a8c304bada6f477c5f31270c2]) successfully deleted
I20260927 22:57:07.692914 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.667305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.8.204.129:40025)
0927 22:57:07.667511 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.667517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.667535 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:57:07.667590 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.667595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.667677 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.667799 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:07.671223 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.672737 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.675573 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.676550 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.676713 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.681575 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:07.681588 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:07.681605 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:07.681654 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:07.686236 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.687198 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.687206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.687211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.687577 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.688334 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.688342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.688348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.688875 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:07.689170 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.692344 (+ 3174us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.692583 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:57:07.693634 25028 catalog_manager.cc:4841] TS ee7ba1b7db0548109c69b9ad8416cbe2 (127.8.204.129:40025): tablet d403d8c1f0764b9c948d02149f69f36c (table test-table [id=ea99311a8c304bada6f477c5f31270c2]) successfully deleted
I20260927 22:57:07.696413 25224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:07.699309 25532 ts_tablet_manager.cc:1918] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:07.699491 25532 log.cc:1196] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/d403d8c1f0764b9c948d02149f69f36c
I20260927 22:57:07.699996 25532 ts_tablet_manager.cc:1939] T d403d8c1f0764b9c948d02149f69f36c P bea7c04cfea94b71a8e499d18633a73d: Deleting consensus metadata
I20260927 22:57:07.701556 25030 catalog_manager.cc:4841] TS bea7c04cfea94b71a8e499d18633a73d (127.8.204.131:39549): tablet d403d8c1f0764b9c948d02149f69f36c (table test-table [id=ea99311a8c304bada6f477c5f31270c2]) successfully deleted
I20260927 22:57:07.811223 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25080
I20260927 22:57:07.830935 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25233
I20260927 22:57:07.844573 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.831805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.8.204.130:38431)
0927 22:57:07.831995 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.831999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.832018 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:57:07.832173 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.832177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.832260 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.832344 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:07.833669 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.834338 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.835744 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.836146 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.836237 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.838911 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:07.838926 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:07.838942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:07.838977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:07.841868 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.842583 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.842590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.842592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.842736 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.843110 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.843114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.843116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.843550 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:07.843775 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.844048 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.844222 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:57:07.845222 25356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:07.937384 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25233
I20260927 22:57:07.954394 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25364
I20260927 22:57:07.968375 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:07.955134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.8.204.131:39549)
0927 22:57:07.955377 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:07.955381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:07.955399 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:57:07.955533 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:07.955537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:07.955599 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:07.955693 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:07.957080 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.957797 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.959080 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:07.959645 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:07.959740 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:07.962169 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:07.962181 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:07.962197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:07.962223 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:07.964776 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.965753 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.965763 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.965769 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.965965 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:07.966413 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:07.966419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:07.966424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:07.966929 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:07.967251 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:07.967653 (+ 402us) server_negotiation.cc:299] Negotiation successful
0927 22:57:07.967911 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:57:07.969103 25487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:08.060039 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25364
I20260927 22:57:08.080358 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25010
I20260927 22:57:08.081388 25073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:08.199918 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25010
I20260927 22:57:28.223515 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40233
--webserver_interface=127.8.204.190
--webserver_port=42221
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:28.551222 25539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:28.551517 25539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:28.551707 25539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:28.558921 25539 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:57:28.559020 25539 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:57:28.559110 25539 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:57:28.559207 25539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:57:28.559319 25539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:28.559441 25539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:28.559502 25539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:57:28.559598 25539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:57:28.569541 25539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:28.570808 25539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:28.572154 25539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:28.584096 25545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:28.584051 25547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:28.584146 25544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:28.585026 25539 server_base.cc:1034] running on GCE node
I20260927 22:57:28.585774 25539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:28.587013 25539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:28.588230 25539 hybrid_clock.cc:648] HybridClock initialized: now 1790549848588157 us; error 50 us; skew 500 ppm
I20260927 22:57:28.588505 25539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:28 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549848, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:28.596290 25539 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:57:28.597819 25539 webserver.cc:460] Webserver started at http://127.8.204.190:42221/ using document root <none> and password file <none>
I20260927 22:57:28.598546 25539 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:28.598680 25539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:28.604488 25539 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:57:28.607724 25554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:28.609117 25539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:57:28.609300 25539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "4e8d418a13ba4357a6fecc6e59cd34d2"
format_stamp: "Formatted at 2026-09-27 22:56:42 on dist-test-slave-40vx"
server_key: "aa7da9650d2b617641b8720f5d165917"
server_key_iv: "fc983e749df514e9c2995a32358a2a4b"
server_key_version: "encryptionkey@0"
I20260927 22:57:28.610288 25539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:28.624590 25539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:28.628886 25539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:28.629191 25539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:28.649566 25539 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:40233
I20260927 22:57:28.649610 25605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:40233 every 8 connection(s)
I20260927 22:57:28.651139 25539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:57:28.656428 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25539
I20260927 22:57:28.657385 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:40025
--local_ip_for_outbound_sockets=127.8.204.129
--tserver_master_addrs=127.8.204.190:40233
--webserver_port=37797
--webserver_interface=127.8.204.129
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:57:28.658916 25606 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:57:28.664161 25606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap starting.
I20260927 22:57:28.685801 25606 log.cc:824] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:28.701181 25606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:57:28.701704 25606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Bootstrap complete.
I20260927 22:57:28.710958 25606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:28.711956 25606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e8d418a13ba4357a6fecc6e59cd34d2, State: Initialized, Role: FOLLOWER
I20260927 22:57:28.712625 25606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:28.712859 25606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:57:28.712996 25606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:57:28.713177 25606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:57:28.717329 25606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:28.717809 25606 leader_election.cc:302] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e8d418a13ba4357a6fecc6e59cd34d2; no voters:
I20260927 22:57:28.718492 25606 leader_election.cc:288] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:57:28.718626 25610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:57:28.719832 25610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [term 3 LEADER]: Becoming Leader. State: Replica: 4e8d418a13ba4357a6fecc6e59cd34d2, State: Running, Role: LEADER
I20260927 22:57:28.720463 25610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } }
I20260927 22:57:28.720804 25606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:57:28.722820 25612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e8d418a13ba4357a6fecc6e59cd34d2. Latest consensus state: current_term: 3 leader_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } } }
I20260927 22:57:28.723179 25612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:28.722945 25611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e8d418a13ba4357a6fecc6e59cd34d2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 40233 } } }
I20260927 22:57:28.723371 25611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:28.732966 25617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:57:28.741631 25617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:57:28.743104 25617 catalog_manager.cc:1259] Loaded cluster ID: fe8a013763e94d51baf95747422a06b7
I20260927 22:57:28.743261 25617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:57:28.747901 25617 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:57:28.749477 25617 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Loaded TSK: 0
I20260927 22:57:28.749938 25617 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e8d418a13ba4357a6fecc6e59cd34d2: Loaded TSK: 1
F20260927 22:57:28.751045 25617 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6575d979ad google::LogMessage::Fail() at ??:0
@ 0x7f6575d9a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6575d974de google::LogMessage::Flush() at ??:0
@ 0x7f6575d98a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6579afb7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6579aea73e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260927 22:57:28.777770 25628 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f658b7c61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f658b7c5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f658b8512a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f658b8510c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f658b89b5e9 std::function<>::operator()() at ??:0
@ 0x7f658b7c9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f658b7c7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f658b84f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f658b84f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f65789e3345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f65789e691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f65789e6799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f65789c1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f657bc9b6db start_thread at ??:0
@ 0x7f6574c3161f clone at ??:0
W20260927 22:57:29.057909 25608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:29.058168 25608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:29.058343 25608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:29.065591 25608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:29.065698 25608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:29.065889 25608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:57:29.075575 25608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:40025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=37797
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:29.076786 25608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:29.078088 25608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:29.090652 25653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:29.090763 25656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:29.090788 25654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:29.091579 25608 server_base.cc:1034] running on GCE node
I20260927 22:57:29.092188 25608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:29.093091 25608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:29.094318 25608 hybrid_clock.cc:648] HybridClock initialized: now 1790549849094271 us; error 44 us; skew 500 ppm
I20260927 22:57:29.094671 25608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:29 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:29.097780 25608 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:57:29.099213 25608 webserver.cc:460] Webserver started at http://127.8.204.129:37797/ using document root <none> and password file <none>
I20260927 22:57:29.099901 25608 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:29.100044 25608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:29.105105 25608 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 22:57:29.107762 25663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:29.108999 25608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:29.109167 25608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "ee7ba1b7db0548109c69b9ad8416cbe2"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "47bc583ed2fb1b336e3263c7137fdd41"
server_key_iv: "e4c71591938d564957bdd1bcd8f438c3"
server_key_version: "encryptionkey@0"
I20260927 22:57:29.110112 25608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:29.151022 25608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:29.155090 25608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:29.155401 25608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:29.156411 25608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:29.157794 25608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:29.157920 25608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:29.158056 25608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:29.158146 25608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:29.192183 25608 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:40025
I20260927 22:57:29.192267 25775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:40025 every 8 connection(s)
I20260927 22:57:29.193674 25608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:57:29.194909 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25608
I20260927 22:57:29.195653 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:38431
--local_ip_for_outbound_sockets=127.8.204.130
--tserver_master_addrs=127.8.204.190:40233
--webserver_port=41895
--webserver_interface=127.8.204.130
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:57:29.198911 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:29.196843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:34125)
0927 22:57:29.197690 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:29.198012 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":352,"threads_started":1}
W20260927 22:57:29.199635 25776 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:40233 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.190:40233: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
I20260927 22:57:29.200485 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:29.200178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:44153)
0927 22:57:29.200317 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:29.200382 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:57:29.201280 25778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:29.200968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:35289)
0927 22:57:29.201102 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:29.201180 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:57:29.201486 25776 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:57:29.524538 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:29.524796 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:29.524886 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:29.531749 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:29.531840 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:29.531945 25779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:57:29.541363 25779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:38431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=41895
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:29.542490 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:29.543699 25779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:29.556314 25785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:29.556386 25787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:29.556488 25784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:29.557385 25779 server_base.cc:1034] running on GCE node
I20260927 22:57:29.558028 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:29.558859 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:29.560055 25779 hybrid_clock.cc:648] HybridClock initialized: now 1790549849560023 us; error 33 us; skew 500 ppm
I20260927 22:57:29.560351 25779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:29 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:29.563318 25779 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:57:29.564620 25779 webserver.cc:460] Webserver started at http://127.8.204.130:41895/ using document root <none> and password file <none>
I20260927 22:57:29.565507 25779 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:29.565661 25779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:29.570909 25779 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:57:29.573597 25794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:29.574893 25779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:57:29.575076 25779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "2e820f2f6cb0446785d9be5325ee63a3"
format_stamp: "Formatted at 2026-09-27 22:56:43 on dist-test-slave-40vx"
server_key: "df0d6584e054d3b3833b34de46814b5d"
server_key_iv: "16b114eb990434f51c2c8a54fac188be"
server_key_version: "encryptionkey@0"
I20260927 22:57:29.576100 25779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:29.599979 25779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:29.603842 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:29.604117 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:29.605105 25779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:29.606503 25779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:29.606631 25779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:29.606763 25779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:29.606849 25779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:29.641701 25779 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38431
I20260927 22:57:29.641785 25906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38431 every 8 connection(s)
I20260927 22:57:29.643111 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:57:29.648109 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:29.646109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:51009)
0927 22:57:29.646844 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:29.647130 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":288,"threads_started":1}
W20260927 22:57:29.648907 25907 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:40233 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.190:40233: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
I20260927 22:57:29.649858 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25779
I20260927 22:57:29.649947 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:29.649513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:40125)
0927 22:57:29.649714 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:29.649806 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 22:57:29.650635 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:39549
--local_ip_for_outbound_sockets=127.8.204.131
--tserver_master_addrs=127.8.204.190:40233
--webserver_port=44501
--webserver_interface=127.8.204.131
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:57:29.650919 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:29.650507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:56531)
0927 22:57:29.650649 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:29.650753 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:57:29.651206 25907 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:57:29.989537 25910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:29.989813 25910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:29.990011 25910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:29.997327 25910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:29.997469 25910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:29.997651 25910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:57:30.007738 25910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:39549
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=44501
--tserver_master_addrs=127.8.204.190:40233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:30.009102 25910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:30.010468 25910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:30.024304 25915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:30.024256 25916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:30.024219 25918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:30.025116 25910 server_base.cc:1034] running on GCE node
I20260927 22:57:30.025674 25910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:30.026640 25910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:30.027851 25910 hybrid_clock.cc:648] HybridClock initialized: now 1790549850027799 us; error 48 us; skew 500 ppm
I20260927 22:57:30.028203 25910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:30 dist-test-slave-40vx krb5kdc[24458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549850, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:30.031378 25910 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:57:30.032789 25910 webserver.cc:460] Webserver started at http://127.8.204.131:44501/ using document root <none> and password file <none>
I20260927 22:57:30.033476 25910 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:30.033622 25910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:30.038993 25910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:30.041738 25925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:30.043027 25910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:57:30.043144 25910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "bea7c04cfea94b71a8e499d18633a73d"
format_stamp: "Formatted at 2026-09-27 22:56:44 on dist-test-slave-40vx"
server_key: "99923efcc04589b31e68157f4c138e1a"
server_key_iv: "728f0dc0f02be10c3bbde35aa8bd96b6"
server_key_version: "encryptionkey@0"
I20260927 22:57:30.044134 25910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:30.058955 25910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:30.062803 25910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:30.063012 25910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:30.063958 25910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:30.065184 25910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:30.065271 25910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:30.065335 25910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:30.065387 25910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:30.097784 25910 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:39549
I20260927 22:57:30.097873 26037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:39549 every 8 connection(s)
I20260927 22:57:30.099212 25910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:57:30.103643 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 25910
I20260927 22:57:30.104209 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:30.102191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:51019)
0927 22:57:30.102915 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:30.103150 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":281,"threads_started":1}
W20260927 22:57:30.104928 26038 heartbeater.cc:644] Failed to heartbeat to 127.8.204.190:40233 (0 consecutive failures): Network error: Failed to ping master at 127.8.204.190:40233: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
I20260927 22:57:30.105799 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:30.105479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:56783)
0927 22:57:30.105624 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:30.105701 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:57:30.106629 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:30.106292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:45697)
0927 22:57:30.106426 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:30.106511 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:57:30.106850 26038 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:57:30.202844 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:30.202037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:54647)
0927 22:57:30.202489 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:30.202616 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":233,"threads_started":1}
I20260927 22:57:30.652741 26043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:30.651840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:44469)
0927 22:57:30.652360 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:30.652501 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
I20260927 22:57:31.108273 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:31.107495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:39051)
0927 22:57:31.107941 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:31.108051 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":212,"threads_started":1}
I20260927 22:57:31.204483 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:31.203788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:47731)
0927 22:57:31.204237 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:31.204321 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":237,"threads_started":1}
I20260927 22:57:31.654510 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:31.653705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:44921)
0927 22:57:31.654267 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:31.654348 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
I20260927 22:57:32.109949 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:32.109210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:46835)
0927 22:57:32.109702 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:32.109787 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":249,"threads_started":1}
I20260927 22:57:32.206051 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:32.205343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:52997)
0927 22:57:32.205769 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:32.205863 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":233,"threads_started":1}
I20260927 22:57:32.656106 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:32.655424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:56137)
0927 22:57:32.655844 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:32.655925 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260927 22:57:33.111650 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:33.110926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:47519)
0927 22:57:33.111416 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:33.111490 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":244,"threads_started":1}
I20260927 22:57:33.207753 26051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:33.206926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:35221)
0927 22:57:33.207505 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:33.207587 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":237,"threads_started":1}
I20260927 22:57:33.657750 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:33.657011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:49161)
0927 22:57:33.657442 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:33.657552 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":203,"threads_started":1}
I20260927 22:57:34.113327 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:34.112556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:50015)
0927 22:57:34.113059 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:34.113135 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":276,"threads_started":1}
I20260927 22:57:34.209404 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:34.208675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:53369)
0927 22:57:34.209138 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:34.209223 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260927 22:57:34.659340 26055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:34.658610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:37595)
0927 22:57:34.659054 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:34.659130 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":230,"threads_started":1}
I20260927 22:57:35.115121 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:35.114370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:58573)
0927 22:57:35.114869 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:35.114958 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260927 22:57:35.211016 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:35.210289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:48485)
0927 22:57:35.210768 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:35.210857 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":268,"threads_started":1}
I20260927 22:57:35.661113 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:35.660264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:52085)
0927 22:57:35.660794 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:35.660899 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":264,"threads_started":1}
I20260927 22:57:36.116642 26059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:36.115960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:49915)
0927 22:57:36.116400 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:36.116488 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260927 22:57:36.212685 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:36.211951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:57943)
0927 22:57:36.212403 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:36.212483 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":204,"threads_started":1}
I20260927 22:57:36.663480 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:36.662194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:49569)
0927 22:57:36.662985 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:36.663108 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":460,"threads_started":1}
I20260927 22:57:37.118407 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:37.117602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:51465)
0927 22:57:37.118090 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:37.118170 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":239,"threads_started":1}
I20260927 22:57:37.214342 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:37.213611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:52391)
0927 22:57:37.214050 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:37.214186 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260927 22:57:37.665292 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:37.664580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:60663)
0927 22:57:37.665044 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:37.665127 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":242,"threads_started":1}
I20260927 22:57:38.120153 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:38.119363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:48513)
0927 22:57:38.119849 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:38.119950 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":230,"threads_started":1}
I20260927 22:57:38.215888 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:38.215143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:41555)
0927 22:57:38.215634 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:38.215722 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":260,"threads_started":1}
I20260927 22:57:38.666998 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:38.666232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:48429)
0927 22:57:38.666737 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:38.666824 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":239,"threads_started":1}
I20260927 22:57:39.121763 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:39.121025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:48593)
0927 22:57:39.121503 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:39.121583 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":223,"threads_started":1}
I20260927 22:57:39.217494 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:39.216800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:49707)
0927 22:57:39.217251 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:39.217337 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":252,"threads_started":1}
I20260927 22:57:39.668735 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:39.668038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:36857)
0927 22:57:39.668469 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:39.668554 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":230,"threads_started":1}
I20260927 22:57:40.123418 26071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:40.122660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:33483)
0927 22:57:40.123137 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:40.123214 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":276,"threads_started":1}
I20260927 22:57:40.219158 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:40.218443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:48947)
0927 22:57:40.218873 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:40.218969 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260927 22:57:40.670686 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:40.669803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:33435)
0927 22:57:40.670375 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:40.670504 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":276,"threads_started":1}
I20260927 22:57:41.125000 26074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:41.124224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:35045)
0927 22:57:41.124721 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:41.124810 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"spinlock_wait_cycles":21888,"thread_start_us":311,"threads_started":1}
I20260927 22:57:41.220894 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:41.220113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:40609)
0927 22:57:41.220631 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:41.220731 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":270,"threads_started":1}
I20260927 22:57:41.672302 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:41.671655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:34013)
0927 22:57:41.672068 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:41.672150 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":210,"threads_started":1}
I20260927 22:57:42.126514 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:42.125865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:55279)
0927 22:57:42.126290 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:42.126366 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":224,"threads_started":1}
I20260927 22:57:42.222702 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:42.221801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:38651)
0927 22:57:42.222266 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:42.222415 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":253,"threads_started":1}
I20260927 22:57:42.674082 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:42.673285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:52501)
0927 22:57:42.673786 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:42.673892 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":241,"threads_started":1}
I20260927 22:57:43.128130 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:43.127397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:55791)
0927 22:57:43.127852 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:43.127949 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":218,"threads_started":1}
I20260927 22:57:43.224418 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:43.223610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:59479)
0927 22:57:43.224174 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:43.224252 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":354,"threads_started":1}
I20260927 22:57:43.675864 26082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:43.675053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:41483)
0927 22:57:43.675589 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:43.675688 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":309,"threads_started":1}
I20260927 22:57:44.129761 26083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:44.128934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:54361)
0927 22:57:44.129444 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:44.129520 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":301,"threads_started":1}
I20260927 22:57:44.226233 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:44.225455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.129:37367)
0927 22:57:44.225953 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:44.226043 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":277,"threads_started":1}
I20260927 22:57:44.677665 26085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:44.676960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.130:35691)
0927 22:57:44.677385 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:44.677492 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":219,"threads_started":1}
I20260927 22:57:45.111006 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25608
I20260927 22:57:45.129563 26086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:45.112512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.8.204.129:40025)
0927 22:57:45.113172 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:45.113186 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:45.113229 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:57:45.115089 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:45.115099 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:45.115627 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:45.115961 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:45.117438 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:45.118719 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:45.120101 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:45.120814 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:45.120970 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:45.123894 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:45.123902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:45.123919 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:45.123956 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:45.126679 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:45.127368 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:45.127375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:45.127381 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:45.127557 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:45.127873 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:45.127876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:45.127878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:45.128379 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:45.128618 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:45.128955 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:57:45.129284 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":360,"threads_started":1}
I20260927 22:57:45.131125 25771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:45.131714 26087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:45.130797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:40233 (local address 127.8.204.131:58201)
0927 22:57:45.131320 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:45.131450 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.190:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":241,"threads_started":1}
I20260927 22:57:45.218343 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25608
I20260927 22:57:45.235821 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25779
I20260927 22:57:45.253136 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:45.236808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.8.204.130:38431)
0927 22:57:45.237273 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:45.237286 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:45.237322 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:57:45.238836 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:45.238845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:45.239261 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:45.239659 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:45.240939 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:45.242220 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:45.243521 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:45.244203 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:45.244337 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:45.246774 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:45.246781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:45.246797 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:45.246834 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:45.250012 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:45.250650 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:45.250656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:45.250664 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:45.250830 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:45.251136 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:45.251139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:45.251141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:45.251833 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:45.252064 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:45.252409 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 22:57:45.252851 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":202,"threads_started":1}
I20260927 22:57:45.254532 25902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:45.338971 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25779
I20260927 22:57:45.356884 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 25910
I20260927 22:57:45.374202 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:45.358043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.8.204.131:39549)
0927 22:57:45.358630 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:45.358644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:45.358684 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:57:45.360389 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:45.360398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:45.360839 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:45.361117 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:45.362349 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:45.363675 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:45.364865 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:45.365497 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:45.365659 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:45.368110 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:45.368117 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:45.368132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:45.368174 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:45.371093 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:45.371888 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:45.371894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:45.371901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:45.372061 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:45.372427 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:45.372430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:45.372433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:45.372940 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:45.373172 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:45.373461 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 22:57:45.373897 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":275,"threads_started":1}
I20260927 22:57:45.375597 26033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:45.459438 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 25910
2026-09-27T22:57:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64422 ms)
[----------] 30 tests from SecurityITest (292502 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:57:45 dist-test-slave-40vx krb5kdc[26095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:57:45 dist-test-slave-40vx krb5kdc[26095](info): set up 2 sockets
Sep 27 22:57:45 dist-test-slave-40vx krb5kdc[26095](info): commencing operation
krb5kdc: starting...
W20260927 22:57:46.571532 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:57:46 dist-test-slave-40vx krb5kdc[26095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:57:46.818079 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:33163
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:47.156489 26108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:47.156873 26108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:47.157042 26108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:47.164529 26108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:57:47.164719 26108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:47.164825 26108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:47.164927 26108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:57:47.165004 26108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:57:47.175082 26108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:47.176592 26108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:47.177971 26108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:47.190104 26113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:47.190140 26116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:47.190169 26114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:47.191344 26108 server_base.cc:1034] running on GCE node
I20260927 22:57:47.192301 26108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:47.194389 26108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:47.195698 26108 hybrid_clock.cc:648] HybridClock initialized: now 1790549867195633 us; error 55 us; skew 500 ppm
I20260927 22:57:47.196060 26108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:47 dist-test-slave-40vx krb5kdc[26095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:47.204205 26108 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:57:47.205914 26108 webserver.cc:460] Webserver started at http://127.8.204.190:40525/ using document root <none> and password file <none>
I20260927 22:57:47.206672 26108 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:47.206831 26108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:47.207170 26108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:47.210467 26108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "0d5078f21aa5434c94628951a274e90c"
format_stamp: "Formatted at 2026-09-27 22:57:47 on dist-test-slave-40vx"
server_key: "73c100acbfa85c1c764e2ce8b4433a96"
server_key_iv: "05de0527f0bc42da867d0e8a4a83ca4c"
server_key_version: "encryptionkey@0"
I20260927 22:57:47.211331 26108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "0d5078f21aa5434c94628951a274e90c"
format_stamp: "Formatted at 2026-09-27 22:57:47 on dist-test-slave-40vx"
server_key: "73c100acbfa85c1c764e2ce8b4433a96"
server_key_iv: "05de0527f0bc42da867d0e8a4a83ca4c"
server_key_version: "encryptionkey@0"
I20260927 22:57:47.217237 26108 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:57:47.220850 26123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:47.222299 26108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:57:47.222502 26108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "0d5078f21aa5434c94628951a274e90c"
format_stamp: "Formatted at 2026-09-27 22:57:47 on dist-test-slave-40vx"
server_key: "73c100acbfa85c1c764e2ce8b4433a96"
server_key_iv: "05de0527f0bc42da867d0e8a4a83ca4c"
server_key_version: "encryptionkey@0"
I20260927 22:57:47.222723 26108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:47.246907 26108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:47.251011 26108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:47.251322 26108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:47.269697 26108 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:33163
I20260927 22:57:47.269738 26174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:33163 every 8 connection(s)
I20260927 22:57:47.271070 26108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:57:47.272372 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26108
I20260927 22:57:47.272620 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:57:47.273502 9010 external_mini_cluster.cc:1428] Setting key 59eb2a86958276365c6406c29e6910bc
I20260927 22:57:47.276213 26175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:57:47 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:47.290612 26175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c: Bootstrap starting.
I20260927 22:57:47.294087 26175 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:47.295207 26175 log.cc:824] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:47.297248 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:47.275642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.8.204.190:33163)
0927 22:57:47.276429 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:47.276441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:47.276480 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:57:47.278210 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:47.278231 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:47.278583 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:47.278910 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:47.280840 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:47.282202 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:47.284280 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:47.284924 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:47.285046 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:47.290083 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:47.290091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:47.290105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:47.290142 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:47.293162 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:47.293977 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:47.293983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:47.293992 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:47.294157 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:47.294564 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:47.294568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:47.294572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:47.295117 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:47.295467 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:47.295883 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:57:47.296316 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":379,"threads_started":1}
I20260927 22:57:47.298424 26175 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c: No bootstrap required, opened a new log
I20260927 22:57:47.306989 26175 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5078f21aa5434c94628951a274e90c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33163 } }
I20260927 22:57:47.307435 26175 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:47.307569 26175 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5078f21aa5434c94628951a274e90c, State: Initialized, Role: FOLLOWER
I20260927 22:57:47.308142 26175 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5078f21aa5434c94628951a274e90c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33163 } }
I20260927 22:57:47.308348 26175 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:57:47.308452 26175 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:57:47.308591 26175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:47.310775 26175 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5078f21aa5434c94628951a274e90c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33163 } }
I20260927 22:57:47.311215 26175 leader_election.cc:302] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d5078f21aa5434c94628951a274e90c; no voters:
I20260927 22:57:47.311808 26175 leader_election.cc:288] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:57:47.311950 26180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:47.312774 26180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [term 1 LEADER]: Becoming Leader. State: Replica: 0d5078f21aa5434c94628951a274e90c, State: Running, Role: LEADER
I20260927 22:57:47.313277 26180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5078f21aa5434c94628951a274e90c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33163 } }
I20260927 22:57:47.314083 26175 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:57:47.315799 26181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d5078f21aa5434c94628951a274e90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5078f21aa5434c94628951a274e90c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33163 } } }
I20260927 22:57:47.316155 26181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:47.316164 26182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5078f21aa5434c94628951a274e90c. Latest consensus state: current_term: 1 leader_uuid: "0d5078f21aa5434c94628951a274e90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d5078f21aa5434c94628951a274e90c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33163 } } }
I20260927 22:57:47.317040 26182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:47.318467 26185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:57:47.323124 26185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:57:47.332186 26185 catalog_manager.cc:1347] Generated new cluster ID: 927b1db410464f8c87fd1d1c74622f9f
I20260927 22:57:47.332309 26185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:57:47.355726 26185 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:57:47.357816 26185 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:57:47.369110 26185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c: Generated new TSK 0
I20260927 22:57:47.370237 26185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:57:47.435678 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33163
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:47.772387 26201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:47.772670 26201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:47.772840 26201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:47.780197 26201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:47.780308 26201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:47.780510 26201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:57:47.790095 26201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:47.791328 26201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:47.792639 26201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:47.806805 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:47.806998 26209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:47.807399 26206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:47.807924 26201 server_base.cc:1034] running on GCE node
I20260927 22:57:47.808790 26201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:47.810097 26201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:47.811393 26201 hybrid_clock.cc:648] HybridClock initialized: now 1790549867811305 us; error 56 us; skew 500 ppm
I20260927 22:57:47.811789 26201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:47 dist-test-slave-40vx krb5kdc[26095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:47.815111 26201 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:57:47.816615 26201 webserver.cc:460] Webserver started at http://127.8.204.129:43451/ using document root <none> and password file <none>
I20260927 22:57:47.817299 26201 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:47.817451 26201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:47.817782 26201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:47.820909 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "eee5dfd5407e42d9b22764d2cad5653c"
format_stamp: "Formatted at 2026-09-27 22:57:47 on dist-test-slave-40vx"
server_key: "cff38e34568d358a1f802e02903557dc"
server_key_iv: "42e5cf4f5374f34747785292c8458cb4"
server_key_version: "encryptionkey@0"
I20260927 22:57:47.821693 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "eee5dfd5407e42d9b22764d2cad5653c"
format_stamp: "Formatted at 2026-09-27 22:57:47 on dist-test-slave-40vx"
server_key: "cff38e34568d358a1f802e02903557dc"
server_key_iv: "42e5cf4f5374f34747785292c8458cb4"
server_key_version: "encryptionkey@0"
I20260927 22:57:47.826911 26201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:57:47.830453 26216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:47.831949 26201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:57:47.832130 26201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "eee5dfd5407e42d9b22764d2cad5653c"
format_stamp: "Formatted at 2026-09-27 22:57:47 on dist-test-slave-40vx"
server_key: "cff38e34568d358a1f802e02903557dc"
server_key_iv: "42e5cf4f5374f34747785292c8458cb4"
server_key_version: "encryptionkey@0"
I20260927 22:57:47.832330 26201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:47.842692 26201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:47.846603 26201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:47.846920 26201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:47.847976 26201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:47.849292 26201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:47.849397 26201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:47.849543 26201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:47.849622 26201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:47.883251 26201 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:39853
I20260927 22:57:47.883381 26328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:39853 every 8 connection(s)
I20260927 22:57:47.884604 26201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:57:47.891925 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26201
I20260927 22:57:47.892221 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:57:47.892922 9010 external_mini_cluster.cc:1428] Setting key e5d9a41e7ca71fa035aa0428ba1f7df6
Sep 27 22:57:47 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:47.906210 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:47.887130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:56347 (local address 127.8.204.190:33163)
0927 22:57:47.887673 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:47.887678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:47.888643 (+ 965us) server_negotiation.cc:407] Connection header received
0927 22:57:47.890509 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:47.890515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:47.890617 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:47.890739 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:47.892498 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:47.893268 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:47.895449 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:47.895856 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:47.895995 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:47.899537 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:47.899546 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:47.899555 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:47.899601 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:47.903130 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:47.903899 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:47.903909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:47.903911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:47.904026 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:47.904465 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:47.904470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:47.904474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:47.904738 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:47.904897 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:47.905605 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 22:57:47.905828 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
I20260927 22:57:47.907155 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:47.887764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33163 (local address 127.8.204.129:56347)
0927 22:57:47.888460 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:47.888506 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:47.890230 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:47.890985 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:47.890993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:47.891645 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:47.892297 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:47.892336 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:47.893412 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:47.893418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:47.895242 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:47.895258 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:47.896095 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:47.896103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:47.896343 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:47.897059 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:47.897082 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:47.899292 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:47.903307 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:47.903318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:47.903336 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:47.903767 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:47.904152 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:47.904157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:47.904160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:47.904343 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:47.904869 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:47.904877 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:47.905213 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 22:57:47.905473 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 22:57:47.906003 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":270,"threads_started":1}
I20260927 22:57:47.908799 26329 heartbeater.cc:347] Connected to a master server at 127.8.204.190:33163
I20260927 22:57:47.909148 26329 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:47.910034 26329 heartbeater.cc:511] Master 127.8.204.190:33163 requested a full tablet report, sending...
I20260927 22:57:47.912595 26140 ts_manager.cc:194] Registered new tserver with Master: eee5dfd5407e42d9b22764d2cad5653c (127.8.204.129:39853)
I20260927 22:57:47.914894 26140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:56347
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:57:47.950110 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33163
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:48.271793 26335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:48.272045 26335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:48.272215 26335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:48.279486 26335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:48.279592 26335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:48.279806 26335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:57:48.289258 26335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:48.290508 26335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:48.291832 26335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:48.304926 26340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:48.304899 26341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:48.305012 26343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:48.306255 26335 server_base.cc:1034] running on GCE node
I20260927 22:57:48.306850 26335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:48.307897 26335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:48.309100 26335 hybrid_clock.cc:648] HybridClock initialized: now 1790549868309040 us; error 49 us; skew 500 ppm
I20260927 22:57:48.309434 26335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:48 dist-test-slave-40vx krb5kdc[26095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:48.312762 26335 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:57:48.314354 26335 webserver.cc:460] Webserver started at http://127.8.204.130:33313/ using document root <none> and password file <none>
I20260927 22:57:48.315070 26335 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:48.315220 26335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:48.315550 26335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:48.318583 26335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "8bc3f08f8521485e9a9e8675a1bed217"
format_stamp: "Formatted at 2026-09-27 22:57:48 on dist-test-slave-40vx"
server_key: "56491113f9b9cfa3cab10b69de582ce5"
server_key_iv: "6118107193676f472182e3d81aef5b95"
server_key_version: "encryptionkey@0"
I20260927 22:57:48.319489 26335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "8bc3f08f8521485e9a9e8675a1bed217"
format_stamp: "Formatted at 2026-09-27 22:57:48 on dist-test-slave-40vx"
server_key: "56491113f9b9cfa3cab10b69de582ce5"
server_key_iv: "6118107193676f472182e3d81aef5b95"
server_key_version: "encryptionkey@0"
I20260927 22:57:48.324611 26335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:57:48.328343 26350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:48.329726 26335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:57:48.329890 26335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "8bc3f08f8521485e9a9e8675a1bed217"
format_stamp: "Formatted at 2026-09-27 22:57:48 on dist-test-slave-40vx"
server_key: "56491113f9b9cfa3cab10b69de582ce5"
server_key_iv: "6118107193676f472182e3d81aef5b95"
server_key_version: "encryptionkey@0"
I20260927 22:57:48.330071 26335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:48.361495 26335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:48.365410 26335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:48.365662 26335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:48.366561 26335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:48.367844 26335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:48.367919 26335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:48.368008 26335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:48.368052 26335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:48.402101 26335 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:35529
I20260927 22:57:48.402149 26462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:35529 every 8 connection(s)
I20260927 22:57:48.403439 26335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:57:48.413734 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26335
I20260927 22:57:48.413945 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:57:48.414592 9010 external_mini_cluster.cc:1428] Setting key 7c633b39d393e589e09b2143f47206cf
Sep 27 22:57:48 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:48.424129 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:48.405942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:42533 (local address 127.8.204.190:33163)
0927 22:57:48.406113 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:48.406117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:48.407370 (+ 1253us) server_negotiation.cc:407] Connection header received
0927 22:57:48.409244 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:48.409249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:48.409315 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:48.409404 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:48.410785 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.411556 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.413034 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.413458 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.413608 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:48.418088 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:48.418095 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:48.418101 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:48.418139 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:48.421086 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:48.421793 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:48.421801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:48.421806 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:48.421975 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:48.422391 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:48.422398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:48.422401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:48.422670 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:48.422840 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:48.423565 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 22:57:48.423793 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:57:48.425137 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:48.406476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33163 (local address 127.8.204.130:42533)
0927 22:57:48.407192 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:48.407229 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:48.408977 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:48.409594 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:48.409601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:48.410118 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:48.410645 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:48.410661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.411702 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.411706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:48.412914 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:48.412926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.413806 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.413815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:48.414276 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:48.415133 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:48.415171 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:48.417868 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:48.421222 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:48.421232 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:48.421250 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:48.421661 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:48.422085 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:48.422090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:48.422094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:48.422280 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:48.422800 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:48.422809 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:48.423153 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 22:57:48.423478 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 22:57:48.424014 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":292,"threads_started":1}
I20260927 22:57:48.426597 26463 heartbeater.cc:347] Connected to a master server at 127.8.204.190:33163
I20260927 22:57:48.426932 26463 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:48.427788 26463 heartbeater.cc:511] Master 127.8.204.190:33163 requested a full tablet report, sending...
I20260927 22:57:48.429260 26140 ts_manager.cc:194] Registered new tserver with Master: 8bc3f08f8521485e9a9e8675a1bed217 (127.8.204.130:35529)
I20260927 22:57:48.430076 26140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:42533
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:57:48.468160 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33163
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:48.801880 26468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:48.802234 26468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:48.802392 26468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:48.809800 26468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:48.809944 26468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:48.810128 26468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:57:48.820117 26468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:48.821444 26468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:48.822789 26468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:48.836063 26473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:48.836115 26474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:48.836297 26476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:48.836922 26468 server_base.cc:1034] running on GCE node
I20260927 22:57:48.837565 26468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:48.838452 26468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:48.839689 26468 hybrid_clock.cc:648] HybridClock initialized: now 1790549868839627 us; error 50 us; skew 500 ppm
I20260927 22:57:48.840011 26468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:48 dist-test-slave-40vx krb5kdc[26095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:48.843113 26468 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:57:48.844590 26468 webserver.cc:460] Webserver started at http://127.8.204.131:40537/ using document root <none> and password file <none>
I20260927 22:57:48.845301 26468 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:48.845451 26468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:48.845773 26468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:48.848901 26468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "0c7c5a8576694580a300dda95ec0a3b1"
format_stamp: "Formatted at 2026-09-27 22:57:48 on dist-test-slave-40vx"
server_key: "85a405d43d0aed26f57e14c47cdc5b09"
server_key_iv: "57a833c21a0d18e40b4f06ffd7f37b4d"
server_key_version: "encryptionkey@0"
I20260927 22:57:48.849730 26468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "0c7c5a8576694580a300dda95ec0a3b1"
format_stamp: "Formatted at 2026-09-27 22:57:48 on dist-test-slave-40vx"
server_key: "85a405d43d0aed26f57e14c47cdc5b09"
server_key_iv: "57a833c21a0d18e40b4f06ffd7f37b4d"
server_key_version: "encryptionkey@0"
I20260927 22:57:48.854874 26468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 22:57:48.858567 26483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:48.859992 26468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 22:57:48.860167 26468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "0c7c5a8576694580a300dda95ec0a3b1"
format_stamp: "Formatted at 2026-09-27 22:57:48 on dist-test-slave-40vx"
server_key: "85a405d43d0aed26f57e14c47cdc5b09"
server_key_iv: "57a833c21a0d18e40b4f06ffd7f37b4d"
server_key_version: "encryptionkey@0"
I20260927 22:57:48.860370 26468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:48.886874 26468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:48.891471 26468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:48.891774 26468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:48.892736 26468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:48.894017 26468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:48.894125 26468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:48.894268 26468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:48.894341 26468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:48.919093 26329 heartbeater.cc:503] Master 127.8.204.190:33163 was elected leader, sending a full tablet report...
I20260927 22:57:48.927868 26468 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:46029
I20260927 22:57:48.927911 26595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:46029 every 8 connection(s)
I20260927 22:57:48.929183 26468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:57:48.932379 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26468
I20260927 22:57:48.932585 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:57:48.933202 9010 external_mini_cluster.cc:1428] Setting key af8e2ffe1720c70cdf543eee56f67123
Sep 27 22:57:48 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:48.950636 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:48.931881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:49193 (local address 127.8.204.190:33163)
0927 22:57:48.932345 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:48.932350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:48.933588 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 22:57:48.935983 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:48.935988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:48.936080 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:48.936224 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:48.938034 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.938863 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.940656 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.941071 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.941169 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:48.944799 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:48.944804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:48.944807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:48.944838 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:48.947824 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:48.948682 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:48.948687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:48.948690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:48.948828 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:48.949199 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:48.949202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:48.949204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:48.949462 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:48.949574 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:48.950227 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 22:57:48.950383 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":267,"threads_started":1}
I20260927 22:57:48.951682 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:48.932556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33163 (local address 127.8.204.131:49193)
0927 22:57:48.933422 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:48.933477 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:48.935711 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:48.936646 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:48.936655 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:48.937279 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:48.937857 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:48.937875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.939017 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.939021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:48.940376 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:48.940389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.941269 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.941274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:48.941479 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:48.942323 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:48.942348 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:48.944601 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:48.947972 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:48.947979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:48.948008 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:48.948544 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:48.948927 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:48.948931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:48.948933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:48.949101 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:48.949573 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:48.949578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:48.949900 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 22:57:48.950121 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 22:57:48.950495 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":297,"threads_started":1}
I20260927 22:57:48.952966 26596 heartbeater.cc:347] Connected to a master server at 127.8.204.190:33163
I20260927 22:57:48.953225 26596 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:48.953857 26596 heartbeater.cc:511] Master 127.8.204.190:33163 requested a full tablet report, sending...
I20260927 22:57:48.955112 26140 ts_manager.cc:194] Registered new tserver with Master: 0c7c5a8576694580a300dda95ec0a3b1 (127.8.204.131:46029)
I20260927 22:57:48.955935 26140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:49193
I20260927 22:57:48.960947 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:57:48.978513 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:48.965598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.8.204.190:33163)
0927 22:57:48.965930 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:48.965938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:48.966294 (+ 356us) server_negotiation.cc:407] Connection header received
0927 22:57:48.966506 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:48.966511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:48.966573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:48.966674 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:48.968223 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.968940 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.970298 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:48.970687 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:48.970834 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:48.973238 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:48.973244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:48.973247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:48.973279 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:48.975865 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:48.976536 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:48.976542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:48.976546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:48.976675 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:48.977039 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:48.977045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:48.977049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:48.977371 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:48.977655 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:48.977948 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:57:48.978183 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260927 22:57:48.982854 26140 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:57:48.989923 26140 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:57:49.026613 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.017102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.8.204.131:46029)
0927 22:57:49.019200 (+ 2098us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.019208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.019243 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:57:49.019366 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.019374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.019596 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.019777 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:49.020626 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.021821 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.024355 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.025299 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.025456 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.025713 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.026033 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.026242 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1928,"thread_start_us":1780,"threads_started":1}
I20260927 22:57:49.026660 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.017005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:46029 (local address 127.0.0.1:34744)
0927 22:57:49.017793 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.017810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:49.017921 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:49.020056 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:49.020061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:49.020096 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:49.020451 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.020462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.021970 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.021976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.024207 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.024220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.025500 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.025506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.025704 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.025823 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 22:57:49.025976 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 22:57:49.026180 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":1,"thread_start_us":523,"threads_started":1}
I20260927 22:57:49.027175 26609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.002395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:35529 (local address 127.0.0.1:49326)
0927 22:57:49.016579 (+ 14184us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.016595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:49.016706 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:49.017786 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:49.017794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:49.017833 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:49.018267 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.018283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.021100 (+ 2817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.021104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.023982 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.023998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.026175 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.026186 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.026391 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.026474 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 22:57:49.026568 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:57:49.026714 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14058,"mutex_wait_us":290,"thread_start_us":12801,"threads_started":1}
I20260927 22:57:49.027415 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.002525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54846 (local address 127.8.204.129:39853)
0927 22:57:49.015631 (+ 13106us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.015646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.017798 (+ 2152us) server_negotiation.cc:407] Connection header received
0927 22:57:49.017945 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.017957 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.018260 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.018474 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:49.019199 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.020796 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.024205 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.025691 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.025810 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.026046 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.026649 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.027028 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12491,"thread_start_us":12748,"threads_started":1}
I20260927 22:57:49.028198 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.002806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.8.204.130:35529)
0927 22:57:49.016128 (+ 13322us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.016135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.016826 (+ 691us) server_negotiation.cc:407] Connection header received
0927 22:57:49.016957 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.016976 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.017276 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.017516 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:49.018441 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.020941 (+ 2500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.024196 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.025698 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.025824 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.026018 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.027426 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.027876 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13158,"thread_start_us":13042,"threads_started":1}
I20260927 22:57:49.028259 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.002332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:54846)
0927 22:57:49.016247 (+ 13915us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.016302 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:49.016596 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:49.018483 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:49.018490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:49.018517 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:49.018973 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.018985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.020926 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.020934 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.023982 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.023998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.026243 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.026248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.026470 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.026495 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 22:57:49.027684 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0927 22:57:49.027831 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12553,"mutex_wait_us":205,"thread_start_us":13960,"threads_started":1}
I20260927 22:57:49.031219 26531 tablet_service.cc:1467] Processing CreateTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 (DEFAULT_TABLE table=test-table [id=e5a845e984da4deaae3267642ade304c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:49.031746 26264 tablet_service.cc:1467] Processing CreateTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 (DEFAULT_TABLE table=test-table [id=e5a845e984da4deaae3267642ade304c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:49.032226 26398 tablet_service.cc:1467] Processing CreateTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 (DEFAULT_TABLE table=test-table [id=e5a845e984da4deaae3267642ade304c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:49.032860 26531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313eacb0e7a0443aa21f25986be76ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:49.033061 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313eacb0e7a0443aa21f25986be76ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:49.034267 26398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 313eacb0e7a0443aa21f25986be76ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:49.044677 26615 tablet_bootstrap.cc:492] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Bootstrap starting.
I20260927 22:57:49.048578 26615 tablet_bootstrap.cc:654] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:49.049737 26617 tablet_bootstrap.cc:492] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Bootstrap starting.
I20260927 22:57:49.049876 26615 log.cc:824] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:49.051586 26616 tablet_bootstrap.cc:492] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Bootstrap starting.
I20260927 22:57:49.054445 26616 tablet_bootstrap.cc:654] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:49.054631 26617 tablet_bootstrap.cc:654] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:49.055855 26616 log.cc:824] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:49.056416 26617 log.cc:824] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:49.059218 26615 tablet_bootstrap.cc:492] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: No bootstrap required, opened a new log
I20260927 22:57:49.059567 26615 ts_tablet_manager.cc:1397] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.002s
I20260927 22:57:49.059707 26617 tablet_bootstrap.cc:492] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: No bootstrap required, opened a new log
I20260927 22:57:49.059986 26617 ts_tablet_manager.cc:1397] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:57:49.060032 26616 tablet_bootstrap.cc:492] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: No bootstrap required, opened a new log
I20260927 22:57:49.060268 26616 ts_tablet_manager.cc:1397] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:57:49.068084 26615 raft_consensus.cc:348] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.068478 26615 raft_consensus.cc:374] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:49.068570 26615 raft_consensus.cc:729] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eee5dfd5407e42d9b22764d2cad5653c, State: Initialized, Role: FOLLOWER
I20260927 22:57:49.069211 26615 consensus_queue.cc:260] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.071053 26615 ts_tablet_manager.cc:1428] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260927 22:57:49.070819 26617 raft_consensus.cc:348] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.070819 26616 raft_consensus.cc:348] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.071254 26617 raft_consensus.cc:374] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:49.071251 26616 raft_consensus.cc:374] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:49.071398 26617 raft_consensus.cc:729] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc3f08f8521485e9a9e8675a1bed217, State: Initialized, Role: FOLLOWER
I20260927 22:57:49.071398 26616 raft_consensus.cc:729] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7c5a8576694580a300dda95ec0a3b1, State: Initialized, Role: FOLLOWER
I20260927 22:57:49.072027 26617 consensus_queue.cc:260] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.072008 26616 consensus_queue.cc:260] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.073179 26596 heartbeater.cc:503] Master 127.8.204.190:33163 was elected leader, sending a full tablet report...
I20260927 22:57:49.073904 26616 ts_tablet_manager.cc:1428] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 22:57:49.074604 26463 heartbeater.cc:503] Master 127.8.204.190:33163 was elected leader, sending a full tablet report...
I20260927 22:57:49.076822 26617 ts_tablet_manager.cc:1428] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
W20260927 22:57:49.136953 26330 tablet.cc:2369] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:57:49.155517 26464 tablet.cc:2369] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:57:49.169727 26623 raft_consensus.cc:482] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:57:49.170071 26623 raft_consensus.cc:504] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.172020 26623 leader_election.cc:288] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eee5dfd5407e42d9b22764d2cad5653c (127.8.204.129:39853), 8bc3f08f8521485e9a9e8675a1bed217 (127.8.204.130:35529)
I20260927 22:57:49.177767 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.172333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:35529 (local address 127.8.204.131:59657)
0927 22:57:49.172572 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.172596 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:49.172729 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:49.173264 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:49.173273 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:49.173311 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:49.173507 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.173516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.174522 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.174529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.175913 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.175925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.177016 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.177022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.177216 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.177240 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 22:57:49.177342 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:57:49.177476 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 22:57:49.177762 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.172823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:59657 (local address 127.8.204.130:35529)
0927 22:57:49.172998 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.173002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.173017 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:57:49.173079 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.173083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.173139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.173241 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:49.173658 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.174381 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.176073 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.176855 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.177022 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.177096 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.177369 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.177495 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:57:49.178668 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.172435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38219 (local address 127.8.204.129:39853)
0927 22:57:49.172647 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.172655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.173394 (+ 739us) server_negotiation.cc:407] Connection header received
0927 22:57:49.173602 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.173606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.173666 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.173780 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:49.174314 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.175021 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.176758 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.177531 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.177725 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.177797 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.178301 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.178428 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:57:49.178710 26624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.172418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.8.204.131:38219)
0927 22:57:49.173280 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.173295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:49.173458 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:49.173794 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:49.173798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:49.173819 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:49.174154 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.174161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.175172 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.175179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.176600 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:49.176612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.177751 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.177761 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:49.178132 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.178169 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:57:49.178271 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:57:49.178391 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":534,"threads_started":1}
I20260927 22:57:49.178845 26418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313eacb0e7a0443aa21f25986be76ae6" candidate_uuid: "0c7c5a8576694580a300dda95ec0a3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc3f08f8521485e9a9e8675a1bed217" is_pre_election: true
I20260927 22:57:49.179497 26418 raft_consensus.cc:2454] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c7c5a8576694580a300dda95ec0a3b1 in term 0.
I20260927 22:57:49.179821 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313eacb0e7a0443aa21f25986be76ae6" candidate_uuid: "0c7c5a8576694580a300dda95ec0a3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee5dfd5407e42d9b22764d2cad5653c" is_pre_election: true
I20260927 22:57:49.180254 26485 leader_election.cc:302] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c7c5a8576694580a300dda95ec0a3b1, 8bc3f08f8521485e9a9e8675a1bed217; no voters:
I20260927 22:57:49.180444 26284 raft_consensus.cc:2454] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c7c5a8576694580a300dda95ec0a3b1 in term 0.
I20260927 22:57:49.180780 26623 raft_consensus.cc:2781] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260927 22:57:49.180807 26597 tablet.cc:2369] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:57:49.181041 26623 raft_consensus.cc:482] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:57:49.181260 26623 raft_consensus.cc:3037] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:49.183671 26623 raft_consensus.cc:504] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.184212 26623 leader_election.cc:288] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [CANDIDATE]: Term 1 election: Requested vote from peers eee5dfd5407e42d9b22764d2cad5653c (127.8.204.129:39853), 8bc3f08f8521485e9a9e8675a1bed217 (127.8.204.130:35529)
I20260927 22:57:49.184660 26284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313eacb0e7a0443aa21f25986be76ae6" candidate_uuid: "0c7c5a8576694580a300dda95ec0a3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eee5dfd5407e42d9b22764d2cad5653c"
I20260927 22:57:49.184679 26418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "313eacb0e7a0443aa21f25986be76ae6" candidate_uuid: "0c7c5a8576694580a300dda95ec0a3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bc3f08f8521485e9a9e8675a1bed217"
I20260927 22:57:49.184851 26284 raft_consensus.cc:3037] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:49.184873 26418 raft_consensus.cc:3037] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:49.186936 26418 raft_consensus.cc:2454] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c7c5a8576694580a300dda95ec0a3b1 in term 1.
I20260927 22:57:49.186987 26284 raft_consensus.cc:2454] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c7c5a8576694580a300dda95ec0a3b1 in term 1.
I20260927 22:57:49.187366 26485 leader_election.cc:302] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c7c5a8576694580a300dda95ec0a3b1, 8bc3f08f8521485e9a9e8675a1bed217; no voters:
I20260927 22:57:49.187659 26623 raft_consensus.cc:2781] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:49.188133 26623 raft_consensus.cc:686] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 1 LEADER]: Becoming Leader. State: Replica: 0c7c5a8576694580a300dda95ec0a3b1, State: Running, Role: LEADER
I20260927 22:57:49.188633 26623 consensus_queue.cc:237] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } }
I20260927 22:57:49.193272 26140 catalog_manager.cc:5495] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c7c5a8576694580a300dda95ec0a3b1 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "0c7c5a8576694580a300dda95ec0a3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c7c5a8576694580a300dda95ec0a3b1" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46029 } health_report { overall_health: HEALTHY } } }
I20260927 22:57:49.219894 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.214329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.8.204.131:46029)
0927 22:57:49.214556 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.214562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.214654 (+ 92us) server_negotiation.cc:407] Connection header received
0927 22:57:49.214872 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.214877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.214952 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.215076 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:57:49.215468 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.216241 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.217528 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.217933 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.218032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.218583 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:57:49.219115 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:57:49.219245 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.219434 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.219533 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:57:49.228101 26284 raft_consensus.cc:1261] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 1 FOLLOWER]: Refusing update from remote peer 0c7c5a8576694580a300dda95ec0a3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:49.228087 26418 raft_consensus.cc:1261] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 1 FOLLOWER]: Refusing update from remote peer 0c7c5a8576694580a300dda95ec0a3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:49.229097 26625 consensus_queue.cc:1035] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eee5dfd5407e42d9b22764d2cad5653c" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:49.229444 26623 consensus_queue.cc:1035] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bc3f08f8521485e9a9e8675a1bed217" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 35529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:49.239833 26628 mvcc.cc:205] Tried to move back new op lower bound from 7334092264347635712 to 7334092264199806976. Current Snapshot: MvccSnapshot[applied={T|T < 7334092264347635712}]
I20260927 22:57:49.254908 26139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:49.255239 26139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:49.259325 26139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d5078f21aa5434c94628951a274e90c: Sending DeleteTablet for 3 replicas of tablet 313eacb0e7a0443aa21f25986be76ae6
I20260927 22:57:49.260457 26531 tablet_service.cc:1514] Processing DeleteTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:49 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:34744
I20260927 22:57:49.260517 26264 tablet_service.cc:1514] Processing DeleteTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:49 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:54846
I20260927 22:57:49.261508 26398 tablet_service.cc:1514] Processing DeleteTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:49 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49326
I20260927 22:57:49.262212 26397 tablet_service.cc:1514] Processing DeleteTablet for tablet 313eacb0e7a0443aa21f25986be76ae6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:49 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49326
I20260927 22:57:49.264864 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26201
I20260927 22:57:49.269291 26640 tablet_replica.cc:330] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: stopping tablet replica
I20260927 22:57:49.269716 26640 raft_consensus.cc:2229] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:49.270150 26640 raft_consensus.cc:2258] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:49.273689 26640 ts_tablet_manager.cc:1905] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:49.285895 26639 tablet_replica.cc:330] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: stopping tablet replica
I20260927 22:57:49.286407 26639 raft_consensus.cc:2229] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:49.286938 26639 raft_consensus.cc:2258] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:49.287786 26638 tablet_replica.cc:330] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: stopping tablet replica
I20260927 22:57:49.288230 26638 raft_consensus.cc:2229] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:57:49.288909 26638 raft_consensus.cc:2258] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:57:49 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:57:49.291186 26638 ts_tablet_manager.cc:1905] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:49.292719 26639 ts_tablet_manager.cc:1905] T 313eacb0e7a0443aa21f25986be76ae6 P eee5dfd5407e42d9b22764d2cad5653c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:49.295616 26640 ts_tablet_manager.cc:1918] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:49.295811 26640 log.cc:1196] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/313eacb0e7a0443aa21f25986be76ae6
I20260927 22:57:49.296411 26640 ts_tablet_manager.cc:1939] T 313eacb0e7a0443aa21f25986be76ae6 P 8bc3f08f8521485e9a9e8675a1bed217: Deleting consensus metadata
I20260927 22:57:49.298045 26638 ts_tablet_manager.cc:1918] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:49.298187 26638 log.cc:1196] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/313eacb0e7a0443aa21f25986be76ae6
I20260927 22:57:49.298525 26125 catalog_manager.cc:4841] TS 8bc3f08f8521485e9a9e8675a1bed217 (127.8.204.130:35529): tablet 313eacb0e7a0443aa21f25986be76ae6 (table test-table [id=e5a845e984da4deaae3267642ade304c]) successfully deleted
I20260927 22:57:49.298643 26638 ts_tablet_manager.cc:1939] T 313eacb0e7a0443aa21f25986be76ae6 P 0c7c5a8576694580a300dda95ec0a3b1: Deleting consensus metadata
I20260927 22:57:49.299211 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.266306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.8.204.129:39853)
0927 22:57:49.266531 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.266541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.277745 (+ 11204us) server_negotiation.cc:407] Connection header received
0927 22:57:49.278281 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.278290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.278402 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.278491 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:49.280820 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.282649 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.285372 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.285948 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.286084 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.290881 (+ 4797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:49.290892 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:49.290904 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:49.290948 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:49.295069 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:49.296266 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:49.296278 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:49.296284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:49.296452 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:49.297027 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:49.297032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:49.297035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:49.297470 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:49.297761 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.298456 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.298820 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:57:49.300212 26124 catalog_manager.cc:4841] TS 0c7c5a8576694580a300dda95ec0a3b1 (127.8.204.131:46029): tablet 313eacb0e7a0443aa21f25986be76ae6 (table test-table [id=e5a845e984da4deaae3267642ade304c]) successfully deleted
W20260927 22:57:49.300292 26125 catalog_manager.cc:4808] TS 8bc3f08f8521485e9a9e8675a1bed217 (127.8.204.130:35529): delete failed for tablet 313eacb0e7a0443aa21f25986be76ae6 because the tablet was not found. No further retry: Not found: Tablet not found: 313eacb0e7a0443aa21f25986be76ae6
I20260927 22:57:49.301292 26324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:49.396010 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26201
W20260927 22:57:49.413610 26124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.204.129:39853 (error 108)
I20260927 22:57:49.413821 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26335
I20260927 22:57:49.415827 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.415338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36020)
0927 22:57:49.415511 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.415643 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:57:49.416095 26124 catalog_manager.cc:4584] TS eee5dfd5407e42d9b22764d2cad5653c (127.8.204.129:39853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 313eacb0e7a0443aa21f25986be76ae6: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Sep 27 22:57:49 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:57:49.430172 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.414999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.8.204.130:35529)
0927 22:57:49.415234 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.415242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.415259 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:57:49.415388 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.415394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.415476 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.415579 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:49.417189 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.418048 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.419586 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.420049 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.420191 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.424103 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:49.424114 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:49.424127 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:49.424156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:49.427261 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:49.428111 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:49.428118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:49.428120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:49.428243 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:49.428624 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:49.428637 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:49.428639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:49.429010 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:49.429323 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.429576 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.429788 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:57:49.430876 26458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:49.469729 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.469277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36024)
0927 22:57:49.469477 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.469574 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:57:49.470825 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.470436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36040)
0927 22:57:49.470623 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.470717 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:57:49.531423 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.530931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36044)
0927 22:57:49.531123 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.531217 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:57:49.532475 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.532147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36052)
0927 22:57:49.532289 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.532369 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:57:49.567948 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26335
I20260927 22:57:49.585635 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26468
Sep 27 22:57:49 dist-test-slave-40vx krb5kdc[26095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:57:49.603704 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.586501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.8.204.131:46029)
0927 22:57:49.586679 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:49.586683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:49.586775 (+ 92us) server_negotiation.cc:407] Connection header received
0927 22:57:49.586949 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:49.586953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:49.587016 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:49.587102 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:49.588935 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.589696 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.591557 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:49.592006 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:49.592110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:49.597346 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:49.597356 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:49.597372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:49.597416 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:49.600830 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:49.601547 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:49.601557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:49.601560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:49.601743 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:49.602123 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:49.602128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:49.602131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:49.602478 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:49.602831 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:49.603066 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:57:49.603314 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:57:49.604451 26591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:49.612898 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.612466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36060)
0927 22:57:49.612656 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.612739 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 22:57:49.613929 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.613554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36062)
0927 22:57:49.613747 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.613827 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:57:49.727420 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26468
I20260927 22:57:49.746320 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26108
I20260927 22:57:49.747593 26170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:49.870280 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:49.869589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39853 (local address 127.0.0.1:36068)
0927 22:57:49.869859 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:49.869976 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.129:39853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 22:57:49.870499 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26108
2026-09-27T22:57:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4409 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:57:49Z Disabled control of system clock
I20260927 22:57:49.941332 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:33365
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:57:50.267067 26650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:50.267357 26650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:50.267441 26650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:50.274686 26650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:57:50.274791 26650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:50.274829 26650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:50.274904 26650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:57:50.274967 26650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:57:50.284183 26650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:33365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:50.285491 26650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:50.287321 26650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:50.298899 26655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:50.298866 26656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:50.299139 26658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:50.299966 26650 server_base.cc:1034] running on GCE node
I20260927 22:57:50.300803 26650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:50.302153 26650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:50.303402 26650 hybrid_clock.cc:648] HybridClock initialized: now 1790549870303352 us; error 37 us; skew 500 ppm
I20260927 22:57:50.303802 26650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:50.306074 26650 webserver.cc:460] Webserver started at http://127.8.204.190:38949/ using document root <none> and password file <none>
I20260927 22:57:50.306785 26650 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:50.306915 26650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:50.307232 26650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:50.310007 26650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "da9a3d1e5d27455dbef34e126be0581c"
format_stamp: "Formatted at 2026-09-27 22:57:50 on dist-test-slave-40vx"
server_key: "024232f22f54f3053b96c3a3c070b9bc"
server_key_iv: "5e7977b5913d0f56daf1558e9daacf92"
server_key_version: "encryptionkey@0"
I20260927 22:57:50.310792 26650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "da9a3d1e5d27455dbef34e126be0581c"
format_stamp: "Formatted at 2026-09-27 22:57:50 on dist-test-slave-40vx"
server_key: "024232f22f54f3053b96c3a3c070b9bc"
server_key_iv: "5e7977b5913d0f56daf1558e9daacf92"
server_key_version: "encryptionkey@0"
I20260927 22:57:50.315660 26650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:57:50.319075 26664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:50.320363 26650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:50.320551 26650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "da9a3d1e5d27455dbef34e126be0581c"
format_stamp: "Formatted at 2026-09-27 22:57:50 on dist-test-slave-40vx"
server_key: "024232f22f54f3053b96c3a3c070b9bc"
server_key_iv: "5e7977b5913d0f56daf1558e9daacf92"
server_key_version: "encryptionkey@0"
I20260927 22:57:50.320799 26650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:50.334439 26650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:50.335201 26650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:50.335536 26650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:50.353113 26650 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:33365
I20260927 22:57:50.353121 26715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:33365 every 8 connection(s)
I20260927 22:57:50.354388 26650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:57:50.357933 26716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:50.364650 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26650
I20260927 22:57:50.364811 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:57:50.365365 9010 external_mini_cluster.cc:1428] Setting key 286818d8057ed92f11bce989ea5a9396
I20260927 22:57:50.368793 26716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c: Bootstrap starting.
I20260927 22:57:50.371826 26716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:50.372857 26716 log.cc:824] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:50.375488 26716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c: No bootstrap required, opened a new log
I20260927 22:57:50.379971 26719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:50.367327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.8.204.190:33365)
0927 22:57:50.368314 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:50.368326 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:50.368371 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:57:50.370317 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:50.370340 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:50.370805 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:50.371207 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:50.371821 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:50.373351 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:50.374892 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:50.375787 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:50.376021 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:50.377260 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:50.377268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:50.377285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:50.378003 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:50.378094 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:50.378117 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:50.378269 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:50.378505 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 22:57:50.378851 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":331,"threads_started":1}
I20260927 22:57:50.383373 26716 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9a3d1e5d27455dbef34e126be0581c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33365 } }
I20260927 22:57:50.383731 26716 raft_consensus.cc:374] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:50.383829 26716 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9a3d1e5d27455dbef34e126be0581c, State: Initialized, Role: FOLLOWER
I20260927 22:57:50.384400 26716 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9a3d1e5d27455dbef34e126be0581c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33365 } }
I20260927 22:57:50.384713 26716 raft_consensus.cc:388] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:57:50.384876 26716 raft_consensus.cc:482] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:57:50.385043 26716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:50.387688 26716 raft_consensus.cc:504] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9a3d1e5d27455dbef34e126be0581c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33365 } }
I20260927 22:57:50.388185 26716 leader_election.cc:302] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da9a3d1e5d27455dbef34e126be0581c; no voters:
I20260927 22:57:50.388976 26716 leader_election.cc:288] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:57:50.389328 26721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:50.390304 26721 raft_consensus.cc:686] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [term 1 LEADER]: Becoming Leader. State: Replica: da9a3d1e5d27455dbef34e126be0581c, State: Running, Role: LEADER
I20260927 22:57:50.390779 26721 consensus_queue.cc:237] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9a3d1e5d27455dbef34e126be0581c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33365 } }
I20260927 22:57:50.391149 26716 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:57:50.393428 26722 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da9a3d1e5d27455dbef34e126be0581c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9a3d1e5d27455dbef34e126be0581c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33365 } } }
I20260927 22:57:50.393886 26722 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:50.394074 26723 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9a3d1e5d27455dbef34e126be0581c. Latest consensus state: current_term: 1 leader_uuid: "da9a3d1e5d27455dbef34e126be0581c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9a3d1e5d27455dbef34e126be0581c" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 33365 } } }
I20260927 22:57:50.394392 26723 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:50.394791 26727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:57:50.401262 26727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:57:50.410055 26727 catalog_manager.cc:1347] Generated new cluster ID: 65224b1def994f44a3c2631000ebe3b4
I20260927 22:57:50.410162 26727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:57:50.423635 26727 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:57:50.425055 26727 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:57:50.431777 26727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c: Generated new TSK 0
I20260927 22:57:50.432797 26727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:57:50.443702 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33365
--builtin_ntp_servers=127.8.204.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:57:50.773343 26740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:50.773635 26740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:50.773722 26740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:50.780793 26740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:50.780884 26740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:50.780987 26740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:57:50.789812 26740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:50.791049 26740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:50.792835 26740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:50.806388 26745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:50.806246 26748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:50.806267 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:50.807029 26740 server_base.cc:1034] running on GCE node
I20260927 22:57:50.807817 26740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:50.808802 26740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:50.810032 26740 hybrid_clock.cc:648] HybridClock initialized: now 1790549870809988 us; error 61 us; skew 500 ppm
I20260927 22:57:50.810365 26740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:50.812620 26740 webserver.cc:460] Webserver started at http://127.8.204.129:44001/ using document root <none> and password file <none>
I20260927 22:57:50.813313 26740 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:50.813446 26740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:50.813766 26740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:50.816727 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "640cd1a134954c5d90ed03b219e19f64"
format_stamp: "Formatted at 2026-09-27 22:57:50 on dist-test-slave-40vx"
server_key: "5543d2997877a244fc0caabdff8beee8"
server_key_iv: "dae2807a7eb03c7e56fba1d54b4158c0"
server_key_version: "encryptionkey@0"
I20260927 22:57:50.817513 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "640cd1a134954c5d90ed03b219e19f64"
format_stamp: "Formatted at 2026-09-27 22:57:50 on dist-test-slave-40vx"
server_key: "5543d2997877a244fc0caabdff8beee8"
server_key_iv: "dae2807a7eb03c7e56fba1d54b4158c0"
server_key_version: "encryptionkey@0"
I20260927 22:57:50.822391 26740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 22:57:50.825934 26754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:50.827255 26740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:57:50.827479 26740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "640cd1a134954c5d90ed03b219e19f64"
format_stamp: "Formatted at 2026-09-27 22:57:50 on dist-test-slave-40vx"
server_key: "5543d2997877a244fc0caabdff8beee8"
server_key_iv: "dae2807a7eb03c7e56fba1d54b4158c0"
server_key_version: "encryptionkey@0"
I20260927 22:57:50.827725 26740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:50.858299 26740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:50.859158 26740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:50.859501 26740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:50.860487 26740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:50.861954 26740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:50.862080 26740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:50.862221 26740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:50.862316 26740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:50.896080 26740 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:44289
I20260927 22:57:50.896130 26866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:44289 every 8 connection(s)
I20260927 22:57:50.897534 26740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:57:50.907105 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26740
I20260927 22:57:50.907279 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:57:50.907835 9010 external_mini_cluster.cc:1428] Setting key 7f69f8b3525d886ed6268097d5a1c4c2
I20260927 22:57:50.910629 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:50.900157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:44807 (local address 127.8.204.190:33365)
0927 22:57:50.900572 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:50.900576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:50.901844 (+ 1268us) server_negotiation.cc:407] Connection header received
0927 22:57:50.903798 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:50.903804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:50.903892 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:50.904009 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:50.905037 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:50.906061 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:50.907833 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:50.908431 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:50.908530 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:50.909934 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:50.909939 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:50.909942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:50.909960 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:50.910000 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:50.910012 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:50.910085 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:50.910328 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:57:50.910400 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":215,"threads_started":1}
I20260927 22:57:50.910768 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33365
--builtin_ntp_servers=127.8.204.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:57:50.912132 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:50.900797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33365 (local address 127.8.204.129:44807)
0927 22:57:50.901688 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:50.901729 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:50.903523 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:50.904237 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:50.904245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:50.904330 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:50.904873 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:50.904888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:50.906263 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:50.906267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:50.907687 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:50.907700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:50.908666 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:50.908673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:50.908874 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:50.909589 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:57:50.909592 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:50.909662 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:57:50.909759 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:50.910115 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:50.910125 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:50.910133 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:57:50.910313 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 22:57:50.910720 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":257,"threads_started":1}
I20260927 22:57:50.913618 26867 heartbeater.cc:347] Connected to a master server at 127.8.204.190:33365
I20260927 22:57:50.913908 26867 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:50.914631 26867 heartbeater.cc:511] Master 127.8.204.190:33365 requested a full tablet report, sending...
I20260927 22:57:50.916827 26681 ts_manager.cc:194] Registered new tserver with Master: 640cd1a134954c5d90ed03b219e19f64 (127.8.204.129:44289)
I20260927 22:57:50.918917 26681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.129:44807
W20260927 22:57:51.236656 26871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:51.236953 26871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:51.237083 26871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:51.244311 26871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:51.244405 26871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:51.244568 26871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:57:51.253572 26871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:51.254859 26871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:51.256590 26871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:51.271353 26879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:51.271363 26877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:51.271411 26876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:51.273144 26871 server_base.cc:1034] running on GCE node
I20260927 22:57:51.274019 26871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:51.275395 26871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:51.276647 26871 hybrid_clock.cc:648] HybridClock initialized: now 1790549871276571 us; error 64 us; skew 500 ppm
I20260927 22:57:51.277096 26871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:51.280068 26871 webserver.cc:460] Webserver started at http://127.8.204.130:36069/ using document root <none> and password file <none>
I20260927 22:57:51.281014 26871 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:51.281168 26871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:51.281553 26871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:51.286360 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "716b5e693261404486f154ca12b2d736"
format_stamp: "Formatted at 2026-09-27 22:57:51 on dist-test-slave-40vx"
server_key: "b440a80f840cd9dc0c59c4169f5dde2d"
server_key_iv: "072890fb80f69236057aa2493911a34b"
server_key_version: "encryptionkey@0"
I20260927 22:57:51.287648 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "716b5e693261404486f154ca12b2d736"
format_stamp: "Formatted at 2026-09-27 22:57:51 on dist-test-slave-40vx"
server_key: "b440a80f840cd9dc0c59c4169f5dde2d"
server_key_iv: "072890fb80f69236057aa2493911a34b"
server_key_version: "encryptionkey@0"
I20260927 22:57:51.295073 26871 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:57:51.298647 26885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:51.300057 26871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:51.300241 26871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "716b5e693261404486f154ca12b2d736"
format_stamp: "Formatted at 2026-09-27 22:57:51 on dist-test-slave-40vx"
server_key: "b440a80f840cd9dc0c59c4169f5dde2d"
server_key_iv: "072890fb80f69236057aa2493911a34b"
server_key_version: "encryptionkey@0"
I20260927 22:57:51.300452 26871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:51.317873 26871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:51.318698 26871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:51.318998 26871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:51.319983 26871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:51.321400 26871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:51.321487 26871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:51.321640 26871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:51.321738 26871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:51.354561 26871 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:33589
I20260927 22:57:51.354590 26997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:33589 every 8 connection(s)
I20260927 22:57:51.356086 26871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:57:51.366436 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 26871
I20260927 22:57:51.366601 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:57:51.367118 9010 external_mini_cluster.cc:1428] Setting key 9e6a8225ae26f3f62673ee3cb577f407
I20260927 22:57:51.369136 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.358675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:51591 (local address 127.8.204.190:33365)
0927 22:57:51.358884 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:51.358891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:51.360508 (+ 1617us) server_negotiation.cc:407] Connection header received
0927 22:57:51.362224 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:51.362230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:51.362308 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:51.362425 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:51.363482 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.364619 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.366149 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.366654 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.366794 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.368304 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:51.368310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:51.368314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:51.368334 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:51.368371 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:51.368386 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:51.368478 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:51.368723 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:57:51.368823 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:57:51.370116 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33365
--builtin_ntp_servers=127.8.204.148:40577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:57:51.370251 27000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.359215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33365 (local address 127.8.204.130:51591)
0927 22:57:51.360312 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:51.360366 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:51.361961 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:51.362617 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:51.362626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:51.362711 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:51.363317 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.363348 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.364742 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.364745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.365982 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.365994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.366875 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.366882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.367103 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.367933 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:57:51.367937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:51.367985 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:57:51.368084 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:51.368489 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:51.368499 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:51.368507 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:57:51.368673 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:57:51.369037 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":327,"threads_started":1}
I20260927 22:57:51.371807 26998 heartbeater.cc:347] Connected to a master server at 127.8.204.190:33365
I20260927 22:57:51.372090 26998 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:51.372787 26998 heartbeater.cc:511] Master 127.8.204.190:33365 requested a full tablet report, sending...
I20260927 22:57:51.374096 26681 ts_manager.cc:194] Registered new tserver with Master: 716b5e693261404486f154ca12b2d736 (127.8.204.130:33589)
I20260927 22:57:51.374913 26681 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.130:51591
W20260927 22:57:51.693049 27001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:51.693331 27001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:51.693456 27001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:51.700898 27001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:51.700986 27001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:51.701151 27001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:57:51.710397 27001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:33365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:51.711664 27001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:51.713456 27001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:51.727900 27006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:51.727967 27007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:51.727934 27009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:51.729171 27001 server_base.cc:1034] running on GCE node
I20260927 22:57:51.729774 27001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:51.730708 27001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:51.731906 27001 hybrid_clock.cc:648] HybridClock initialized: now 1790549871731866 us; error 33 us; skew 500 ppm
I20260927 22:57:51.732363 27001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:51.734735 27001 webserver.cc:460] Webserver started at http://127.8.204.131:34005/ using document root <none> and password file <none>
I20260927 22:57:51.735503 27001 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:51.735621 27001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:51.735953 27001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:51.738757 27001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "57afbf434efd45ee87e6c2ff8f96e339"
format_stamp: "Formatted at 2026-09-27 22:57:51 on dist-test-slave-40vx"
server_key: "98d23c987666148ef33fe491bef7a88c"
server_key_iv: "54002c309a06299488de65b641f8ceef"
server_key_version: "encryptionkey@0"
I20260927 22:57:51.739588 27001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "57afbf434efd45ee87e6c2ff8f96e339"
format_stamp: "Formatted at 2026-09-27 22:57:51 on dist-test-slave-40vx"
server_key: "98d23c987666148ef33fe491bef7a88c"
server_key_iv: "54002c309a06299488de65b641f8ceef"
server_key_version: "encryptionkey@0"
I20260927 22:57:51.744432 27001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:57:51.747802 27015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:51.749025 27001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:57:51.749194 27001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "57afbf434efd45ee87e6c2ff8f96e339"
format_stamp: "Formatted at 2026-09-27 22:57:51 on dist-test-slave-40vx"
server_key: "98d23c987666148ef33fe491bef7a88c"
server_key_iv: "54002c309a06299488de65b641f8ceef"
server_key_version: "encryptionkey@0"
I20260927 22:57:51.749409 27001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:51.762583 27001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:51.763417 27001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:51.763703 27001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:51.764611 27001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:51.765956 27001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:51.766083 27001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:51.766218 27001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:51.766309 27001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:51.799629 27001 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:37593
I20260927 22:57:51.799681 27127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:37593 every 8 connection(s)
I20260927 22:57:51.800890 27001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:57:51.804023 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27001
I20260927 22:57:51.804171 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:57:51.804746 9010 external_mini_cluster.cc:1428] Setting key b2f816b25c4c3ea4d915cebb94dd82a6
I20260927 22:57:51.815596 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.803368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:33231 (local address 127.8.204.190:33365)
0927 22:57:51.803552 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:51.803557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:51.805290 (+ 1733us) server_negotiation.cc:407] Connection header received
0927 22:57:51.807066 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:51.807074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:51.807169 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:51.807308 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:51.809002 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.810296 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.812712 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.813410 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.813548 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.814923 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:51.814927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:51.814931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:51.814949 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:51.814980 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:51.814992 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:51.815059 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:51.815236 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 22:57:51.815357 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:57:51.816641 27130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.803962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:33365 (local address 127.8.204.131:33231)
0927 22:57:51.805120 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:51.805180 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:51.806797 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:51.807779 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:51.807791 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:51.807919 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:51.808783 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.808813 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.810540 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.810547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.812548 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.812563 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.813693 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.813699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.813973 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.814609 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:57:51.814611 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:51.814651 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:57:51.814730 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:51.815058 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:51.815063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:51.815068 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:57:51.815197 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 22:57:51.815571 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":337,"threads_started":1}
I20260927 22:57:51.817998 27128 heartbeater.cc:347] Connected to a master server at 127.8.204.190:33365
I20260927 22:57:51.818305 27128 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:51.819028 27128 heartbeater.cc:511] Master 127.8.204.190:33365 requested a full tablet report, sending...
I20260927 22:57:51.820520 26680 ts_manager.cc:194] Registered new tserver with Master: 57afbf434efd45ee87e6c2ff8f96e339 (127.8.204.131:37593)
I20260927 22:57:51.821671 26680 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.131:33231
I20260927 22:57:51.821841 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:57:51.832708 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.826557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.8.204.190:33365)
0927 22:57:51.826842 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:51.826852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:51.827183 (+ 331us) server_negotiation.cc:407] Connection header received
0927 22:57:51.827399 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:51.827408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:51.827538 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:51.827734 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:51.828166 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.828935 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.830261 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.830684 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.830781 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.831951 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:51.831956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:51.831960 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:51.831979 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:51.832012 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:51.832025 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:51.832111 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:51.832205 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 22:57:51.832282 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:57:51.836649 26680 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:57:51.844530 26680 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:57:51.870142 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.857947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:37593 (local address 127.0.0.1:35228)
0927 22:57:51.860711 (+ 2764us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:51.860737 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:51.860890 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:51.861656 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:51.861663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:51.861704 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:51.862114 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.862128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.864297 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.864303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.866963 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.866980 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.869037 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.869044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.869275 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.869338 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 22:57:51.869509 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 22:57:51.869719 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2594,"mutex_wait_us":66,"thread_start_us":2414,"threads_started":1}
I20260927 22:57:51.870142 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.858460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.8.204.131:37593)
0927 22:57:51.860161 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:51.860175 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:51.861030 (+ 855us) server_negotiation.cc:407] Connection header received
0927 22:57:51.861154 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:51.861166 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:51.861426 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:51.861640 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:51.862329 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.864110 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.867230 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.868830 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.868980 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.869301 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:51.869552 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:57:51.869729 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":1339,"threads_started":1}
I20260927 22:57:51.871356 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.859833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:33589 (local address 127.0.0.1:41216)
0927 22:57:51.861009 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:51.861025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:51.861139 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:51.864363 (+ 3224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:51.864368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:51.864401 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:51.864887 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.864898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.866391 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.866399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.868257 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.868270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.870436 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.870440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.870670 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.870735 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 22:57:51.870821 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 22:57:51.870958 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":74,"thread_start_us":865,"threads_started":1}
I20260927 22:57:51.871488 27142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.857985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.8.204.129:44289)
0927 22:57:51.862330 (+ 4345us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:51.862340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:51.862370 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:57:51.862487 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:51.862497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:51.862769 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:51.862983 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:51.864176 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.865605 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.868613 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.869978 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.870087 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.870284 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:51.870937 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 22:57:51.871101 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4167,"thread_start_us":4056,"threads_started":1}
I20260927 22:57:51.872079 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.857645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44289 (local address 127.0.0.1:37306)
0927 22:57:51.858732 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:51.858805 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:51.859024 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:51.862993 (+ 3969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:51.863001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:51.863037 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:51.863967 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.863979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.865753 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.865759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.868417 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:51.868433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.870433 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.870439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:51.870665 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.870746 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 22:57:51.870821 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 22:57:51.871311 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":428,"thread_start_us":556,"threads_started":1}
I20260927 22:57:51.872867 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:51.860324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.8.204.130:33589)
0927 22:57:51.863556 (+ 3232us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:51.863576 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:51.863603 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:57:51.863696 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:51.863703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:51.863890 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:51.864066 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:51.865086 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.866256 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.868492 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:51.869906 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:51.870057 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:51.870325 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:51.872337 (+ 2012us) server_negotiation.cc:299] Negotiation successful
0927 22:57:51.872527 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3071,"thread_start_us":2946,"threads_started":1}
I20260927 22:57:51.875090 27063 tablet_service.cc:1467] Processing CreateTablet for tablet bc74defe714343f29871d1cc8f03091b (DEFAULT_TABLE table=test-table [id=ceb7f065cf974ad0887b1d0ef2a3d256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:51.875368 26802 tablet_service.cc:1467] Processing CreateTablet for tablet bc74defe714343f29871d1cc8f03091b (DEFAULT_TABLE table=test-table [id=ceb7f065cf974ad0887b1d0ef2a3d256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:51.876547 26802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc74defe714343f29871d1cc8f03091b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:51.876160 26933 tablet_service.cc:1467] Processing CreateTablet for tablet bc74defe714343f29871d1cc8f03091b (DEFAULT_TABLE table=test-table [id=ceb7f065cf974ad0887b1d0ef2a3d256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:51.876673 27063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc74defe714343f29871d1cc8f03091b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:51.877373 26933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc74defe714343f29871d1cc8f03091b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:51.888617 27146 tablet_bootstrap.cc:492] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Bootstrap starting.
I20260927 22:57:51.890760 27147 tablet_bootstrap.cc:492] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Bootstrap starting.
I20260927 22:57:51.892444 27148 tablet_bootstrap.cc:492] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Bootstrap starting.
I20260927 22:57:51.892738 27146 tablet_bootstrap.cc:654] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:51.894057 27146 log.cc:824] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:51.894174 27147 tablet_bootstrap.cc:654] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:51.895447 27148 tablet_bootstrap.cc:654] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:51.895988 27147 log.cc:824] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:51.896597 27148 log.cc:824] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:51.898303 27146 tablet_bootstrap.cc:492] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: No bootstrap required, opened a new log
I20260927 22:57:51.898547 27146 ts_tablet_manager.cc:1397] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 22:57:51.898970 27148 tablet_bootstrap.cc:492] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: No bootstrap required, opened a new log
I20260927 22:57:51.899209 27148 ts_tablet_manager.cc:1397] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:57:51.904812 27147 tablet_bootstrap.cc:492] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: No bootstrap required, opened a new log
I20260927 22:57:51.905051 27147 ts_tablet_manager.cc:1397] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260927 22:57:51.907213 27148 raft_consensus.cc:348] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:51.907711 27148 raft_consensus.cc:374] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:51.907816 27148 raft_consensus.cc:729] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 716b5e693261404486f154ca12b2d736, State: Initialized, Role: FOLLOWER
I20260927 22:57:51.908491 27148 consensus_queue.cc:260] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:51.911847 26998 heartbeater.cc:503] Master 127.8.204.190:33365 was elected leader, sending a full tablet report...
I20260927 22:57:51.912503 27148 ts_tablet_manager.cc:1428] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260927 22:57:51.912429 27146 raft_consensus.cc:348] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:51.912943 27146 raft_consensus.cc:374] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:51.913074 27146 raft_consensus.cc:729] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 640cd1a134954c5d90ed03b219e19f64, State: Initialized, Role: FOLLOWER
I20260927 22:57:51.913668 27146 consensus_queue.cc:260] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:51.914781 26867 heartbeater.cc:503] Master 127.8.204.190:33365 was elected leader, sending a full tablet report...
I20260927 22:57:51.914815 27147 raft_consensus.cc:348] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:51.915234 27147 raft_consensus.cc:374] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:51.915349 27147 raft_consensus.cc:729] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57afbf434efd45ee87e6c2ff8f96e339, State: Initialized, Role: FOLLOWER
I20260927 22:57:51.915452 27146 ts_tablet_manager.cc:1428] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260927 22:57:51.915973 27147 consensus_queue.cc:260] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:51.917047 27128 heartbeater.cc:503] Master 127.8.204.190:33365 was elected leader, sending a full tablet report...
I20260927 22:57:51.917747 27147 ts_tablet_manager.cc:1428] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
W20260927 22:57:52.052526 27129 tablet.cc:2369] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:57:52.108235 26999 tablet.cc:2369] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:57:52.149999 26868 tablet.cc:2369] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:57:52.226876 27152 raft_consensus.cc:482] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:57:52.227231 27152 raft_consensus.cc:504] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:52.228927 27152 leader_election.cc:288] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 640cd1a134954c5d90ed03b219e19f64 (127.8.204.129:44289), 57afbf434efd45ee87e6c2ff8f96e339 (127.8.204.131:37593)
I20260927 22:57:52.238075 27142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.229377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:42807 (local address 127.8.204.129:44289)
0927 22:57:52.229586 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:52.229592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:52.230286 (+ 694us) server_negotiation.cc:407] Connection header received
0927 22:57:52.230520 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:52.230525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:52.230659 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:52.230749 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:52.231649 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.232829 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.235671 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.236907 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.237085 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.237159 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:52.237680 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 22:57:52.237808 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:57:52.238092 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.229385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:40321 (local address 127.8.204.131:37593)
0927 22:57:52.229585 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:52.229592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:52.230370 (+ 778us) server_negotiation.cc:407] Connection header received
0927 22:57:52.230542 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:52.230547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:52.230624 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:52.230737 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:52.231634 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.232779 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.235654 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.236907 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.237092 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.237158 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:52.237689 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 22:57:52.237809 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:57:52.238211 27155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.229353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:37593 (local address 127.8.204.130:40321)
0927 22:57:52.230247 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:52.230262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:52.230408 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:52.230779 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:52.230784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:52.230823 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:52.231408 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:52.231427 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.232929 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.232934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:52.235417 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:52.235429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.237127 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.237131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:52.237446 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.237537 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 22:57:52.237631 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 22:57:52.237820 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":207,"spinlock_wait_cycles":50304,"thread_start_us":422,"threads_started":1}
I20260927 22:57:52.238420 27156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.229266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:44289 (local address 127.8.204.130:42807)
0927 22:57:52.230153 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:52.230176 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:52.230350 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:52.230779 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:52.230784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:52.230813 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:52.231433 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:52.231442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.232949 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.232955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:52.235414 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:52.235429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.237125 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.237131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:52.237454 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.237543 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 22:57:52.237631 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:57:52.237820 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":328,"thread_start_us":550,"threads_started":1}
I20260927 22:57:52.239253 27083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74defe714343f29871d1cc8f03091b" candidate_uuid: "716b5e693261404486f154ca12b2d736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57afbf434efd45ee87e6c2ff8f96e339" is_pre_election: true
I20260927 22:57:52.239454 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74defe714343f29871d1cc8f03091b" candidate_uuid: "716b5e693261404486f154ca12b2d736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "640cd1a134954c5d90ed03b219e19f64" is_pre_election: true
I20260927 22:57:52.239892 27083 raft_consensus.cc:2454] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716b5e693261404486f154ca12b2d736 in term 0.
I20260927 22:57:52.239933 26822 raft_consensus.cc:2454] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 716b5e693261404486f154ca12b2d736 in term 0.
I20260927 22:57:52.240574 26887 leader_election.cc:302] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 57afbf434efd45ee87e6c2ff8f96e339, 640cd1a134954c5d90ed03b219e19f64, 716b5e693261404486f154ca12b2d736; no voters:
I20260927 22:57:52.240958 27152 raft_consensus.cc:2781] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:57:52.241114 27152 raft_consensus.cc:482] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:57:52.241233 27152 raft_consensus.cc:3037] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:52.243402 27152 raft_consensus.cc:504] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:52.244014 27152 leader_election.cc:288] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [CANDIDATE]: Term 1 election: Requested vote from peers 640cd1a134954c5d90ed03b219e19f64 (127.8.204.129:44289), 57afbf434efd45ee87e6c2ff8f96e339 (127.8.204.131:37593)
I20260927 22:57:52.244460 26822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74defe714343f29871d1cc8f03091b" candidate_uuid: "716b5e693261404486f154ca12b2d736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "640cd1a134954c5d90ed03b219e19f64"
I20260927 22:57:52.244654 26822 raft_consensus.cc:3037] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:52.244836 27083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc74defe714343f29871d1cc8f03091b" candidate_uuid: "716b5e693261404486f154ca12b2d736" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57afbf434efd45ee87e6c2ff8f96e339"
I20260927 22:57:52.245041 27083 raft_consensus.cc:3037] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:52.247933 26822 raft_consensus.cc:2454] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716b5e693261404486f154ca12b2d736 in term 1.
I20260927 22:57:52.248162 27083 raft_consensus.cc:2454] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 716b5e693261404486f154ca12b2d736 in term 1.
I20260927 22:57:52.248343 26889 leader_election.cc:302] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 640cd1a134954c5d90ed03b219e19f64, 716b5e693261404486f154ca12b2d736; no voters:
I20260927 22:57:52.248622 27152 raft_consensus.cc:2781] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:52.249040 27152 raft_consensus.cc:686] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 1 LEADER]: Becoming Leader. State: Replica: 716b5e693261404486f154ca12b2d736, State: Running, Role: LEADER
I20260927 22:57:52.249531 27152 consensus_queue.cc:237] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } }
I20260927 22:57:52.253968 26679 catalog_manager.cc:5495] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 reported cstate change: term changed from 0 to 1, leader changed from <none> to 716b5e693261404486f154ca12b2d736 (127.8.204.130). New cstate: current_term: 1 leader_uuid: "716b5e693261404486f154ca12b2d736" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "716b5e693261404486f154ca12b2d736" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 33589 } health_report { overall_health: HEALTHY } } }
I20260927 22:57:52.320456 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.315746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41222 (local address 127.8.204.130:33589)
0927 22:57:52.315921 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:52.315925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:52.315942 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:57:52.316024 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:52.316027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:52.316108 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:52.316236 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:57:52.316537 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.317272 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.318667 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.319057 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.319144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.319572 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:57:52.320053 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:57:52.320175 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:52.320212 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 22:57:52.320267 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:57:52.328668 27083 raft_consensus.cc:1261] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 1 FOLLOWER]: Refusing update from remote peer 716b5e693261404486f154ca12b2d736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:52.328706 26822 raft_consensus.cc:1261] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 1 FOLLOWER]: Refusing update from remote peer 716b5e693261404486f154ca12b2d736: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:52.329821 27157 consensus_queue.cc:1035] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57afbf434efd45ee87e6c2ff8f96e339" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:52.330181 27152 consensus_queue.cc:1035] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [LEADER]: Connected to new peer: Peer: permanent_uuid: "640cd1a134954c5d90ed03b219e19f64" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:52.340830 27160 mvcc.cc:205] Tried to move back new op lower bound from 7334092277048139776 to 7334092276737552384. Current Snapshot: MvccSnapshot[applied={T|T < 7334092277048139776}]
I20260927 22:57:52.347474 27161 mvcc.cc:205] Tried to move back new op lower bound from 7334092277048139776 to 7334092276737552384. Current Snapshot: MvccSnapshot[applied={T|T < 7334092277048139776}]
I20260927 22:57:52.354969 26681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:52.355209 26681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:52.359019 26681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da9a3d1e5d27455dbef34e126be0581c: Sending DeleteTablet for 3 replicas of tablet bc74defe714343f29871d1cc8f03091b
I20260927 22:57:52.364279 26802 tablet_service.cc:1514] Processing DeleteTablet for tablet bc74defe714343f29871d1cc8f03091b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:52 UTC) from {username='slave'} at 127.0.0.1:37306
I20260927 22:57:52.366941 27063 tablet_service.cc:1514] Processing DeleteTablet for tablet bc74defe714343f29871d1cc8f03091b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:52 UTC) from {username='slave'} at 127.0.0.1:35228
I20260927 22:57:52.374812 26933 tablet_service.cc:1514] Processing DeleteTablet for tablet bc74defe714343f29871d1cc8f03091b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:52 UTC) from {username='slave'} at 127.0.0.1:41216
I20260927 22:57:52.381882 27172 tablet_replica.cc:330] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: stopping tablet replica
I20260927 22:57:52.382352 27172 raft_consensus.cc:2229] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:57:52.383060 27172 raft_consensus.cc:2258] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:52.383838 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26740
I20260927 22:57:52.384833 27171 tablet_replica.cc:330] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: stopping tablet replica
I20260927 22:57:52.385267 27171 raft_consensus.cc:2229] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:52.385772 27171 raft_consensus.cc:2258] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:52.386977 27172 ts_tablet_manager.cc:1905] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:52.388962 27170 tablet_replica.cc:330] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: stopping tablet replica
I20260927 22:57:52.389406 27170 raft_consensus.cc:2229] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:52.389910 27170 raft_consensus.cc:2258] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:52.392540 27171 ts_tablet_manager.cc:1905] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:52.397295 27170 ts_tablet_manager.cc:1905] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:52.399293 27172 ts_tablet_manager.cc:1918] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:52.399536 27172 log.cc:1196] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/bc74defe714343f29871d1cc8f03091b
I20260927 22:57:52.400270 27172 ts_tablet_manager.cc:1939] T bc74defe714343f29871d1cc8f03091b P 716b5e693261404486f154ca12b2d736: Deleting consensus metadata
I20260927 22:57:52.402453 26668 catalog_manager.cc:4841] TS 716b5e693261404486f154ca12b2d736 (127.8.204.130:33589): tablet bc74defe714343f29871d1cc8f03091b (table test-table [id=ceb7f065cf974ad0887b1d0ef2a3d256]) successfully deleted
I20260927 22:57:52.404260 27171 ts_tablet_manager.cc:1918] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:52.404444 27171 log.cc:1196] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/bc74defe714343f29871d1cc8f03091b
I20260927 22:57:52.404932 27171 ts_tablet_manager.cc:1939] T bc74defe714343f29871d1cc8f03091b P 57afbf434efd45ee87e6c2ff8f96e339: Deleting consensus metadata
I20260927 22:57:52.406424 27170 ts_tablet_manager.cc:1918] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:52.406606 27170 log.cc:1196] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/bc74defe714343f29871d1cc8f03091b
I20260927 22:57:52.406940 26666 catalog_manager.cc:4841] TS 57afbf434efd45ee87e6c2ff8f96e339 (127.8.204.131:37593): tablet bc74defe714343f29871d1cc8f03091b (table test-table [id=ceb7f065cf974ad0887b1d0ef2a3d256]) successfully deleted
I20260927 22:57:52.407712 27142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.386631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.8.204.129:44289)
0927 22:57:52.386825 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:52.386832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:52.395691 (+ 8859us) server_negotiation.cc:407] Connection header received
0927 22:57:52.396026 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:52.396035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:52.396144 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:52.396226 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:52.396886 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.398040 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.400012 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.400645 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.400756 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.405846 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:52.405858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:52.405877 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:52.406723 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:52.406788 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:52.406814 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:52.406906 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:52.407144 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:57:52.407306 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:57:52.407956 27170 ts_tablet_manager.cc:1939] T bc74defe714343f29871d1cc8f03091b P 640cd1a134954c5d90ed03b219e19f64: Deleting consensus metadata
I20260927 22:57:52.408504 26862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:52.498147 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26740
I20260927 22:57:52.498266 26668 catalog_manager.cc:4841] TS 640cd1a134954c5d90ed03b219e19f64 (127.8.204.129:44289): tablet bc74defe714343f29871d1cc8f03091b (table test-table [id=ceb7f065cf974ad0887b1d0ef2a3d256]) successfully deleted
I20260927 22:57:52.516105 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26871
I20260927 22:57:52.523998 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.517120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41236 (local address 127.8.204.130:33589)
0927 22:57:52.517290 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:52.517294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:52.517311 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:57:52.517367 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:52.517371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:52.517448 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:52.517546 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:52.517990 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.518784 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.520802 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.521256 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.521364 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.522520 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:52.522528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:52.522541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:52.523119 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:52.523164 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:52.523183 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:52.523352 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:52.523490 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 22:57:52.523587 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:57:52.524767 26993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:52.638562 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26871
I20260927 22:57:52.656615 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27001
I20260927 22:57:52.663017 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:52.657431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.8.204.131:37593)
0927 22:57:52.657602 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:52.657606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:52.657624 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:57:52.657799 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:52.657802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:52.657878 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:52.657960 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:52.658316 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.659023 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.660302 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:52.660683 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:52.660781 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:52.661677 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:52.661686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:52.661701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:52.662249 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:52.662298 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:52.662317 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:52.662404 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:52.662562 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 22:57:52.662646 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:57:52.663710 27123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:52.752115 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27001
I20260927 22:57:52.768222 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 26650
I20260927 22:57:52.769300 26711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:52.879400 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 26650
2026-09-27T22:57:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3010 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:57:52 dist-test-slave-40vx krb5kdc[27180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:57:52 dist-test-slave-40vx krb5kdc[27180](info): set up 2 sockets
Sep 27 22:57:52 dist-test-slave-40vx krb5kdc[27180](info): commencing operation
krb5kdc: starting...
W20260927 22:57:53.971387 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:57:54 dist-test-slave-40vx krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:57:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:57:54.208171 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44025
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:54.537961 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:54.538292 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:54.538443 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:54.545523 27193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:57:54.545673 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:54.545773 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:54.545873 27193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:57:54.545945 27193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:57:54.555426 27193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:54.556684 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:54.557907 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:54.569542 27201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:54.569609 27199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:54.569621 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:54.572634 27193 server_base.cc:1034] running on GCE node
I20260927 22:57:54.573400 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:54.574774 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:54.575995 27193 hybrid_clock.cc:648] HybridClock initialized: now 1790549874575957 us; error 47 us; skew 500 ppm
I20260927 22:57:54.576334 27193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:54 dist-test-slave-40vx krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549874, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:54.579392 27193 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:57:54.580942 27193 webserver.cc:460] Webserver started at http://127.8.204.190:32905/ using document root <none> and password file <none>
I20260927 22:57:54.581630 27193 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:54.581782 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:54.582085 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:54.585073 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "3802047216ae4349969d808c50459ba3"
format_stamp: "Formatted at 2026-09-27 22:57:54 on dist-test-slave-40vx"
server_key: "a9ba96d1e97af1184b519e90bd15ca7d"
server_key_iv: "82ccd429267ffed3bda114f9351731f5"
server_key_version: "encryptionkey@0"
I20260927 22:57:54.585832 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "3802047216ae4349969d808c50459ba3"
format_stamp: "Formatted at 2026-09-27 22:57:54 on dist-test-slave-40vx"
server_key: "a9ba96d1e97af1184b519e90bd15ca7d"
server_key_iv: "82ccd429267ffed3bda114f9351731f5"
server_key_version: "encryptionkey@0"
I20260927 22:57:54.590937 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:57:54.594295 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:54.595660 27193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:54.595846 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "3802047216ae4349969d808c50459ba3"
format_stamp: "Formatted at 2026-09-27 22:57:54 on dist-test-slave-40vx"
server_key: "a9ba96d1e97af1184b519e90bd15ca7d"
server_key_iv: "82ccd429267ffed3bda114f9351731f5"
server_key_version: "encryptionkey@0"
I20260927 22:57:54.596040 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:54.615931 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:54.619707 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:54.619992 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:54.638067 27193 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:44025
I20260927 22:57:54.638090 27259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:44025 every 8 connection(s)
I20260927 22:57:54.639382 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:57:54.641443 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27193
I20260927 22:57:54.641636 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:57:54.642071 9010 external_mini_cluster.cc:1428] Setting key 8390bcfbc350db32617bb4ba973fe057
I20260927 22:57:54.643178 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:57:54 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:54.656937 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3: Bootstrap starting.
I20260927 22:57:54.660924 27260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:54.662192 27260 log.cc:824] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:54.664958 27260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3: No bootstrap required, opened a new log
I20260927 22:57:54.665810 27263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:54.643842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.8.204.190:44025)
0927 22:57:54.644627 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:54.644643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:54.644693 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:57:54.646593 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:54.646619 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:54.647131 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:54.647604 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:54.648841 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:54.650858 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:54.652308 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:54.653332 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:54.653532 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:54.657787 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:54.657796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:54.657813 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:54.657855 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:54.661745 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:54.662467 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:54.662474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:54.662479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:54.662642 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:54.663045 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:54.663048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:54.663050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:54.663592 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:54.663904 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:54.664342 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 22:57:54.664771 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":324,"threads_started":1}
I20260927 22:57:54.672916 27260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3802047216ae4349969d808c50459ba3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44025 } }
I20260927 22:57:54.673276 27260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:54.673385 27260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3802047216ae4349969d808c50459ba3, State: Initialized, Role: FOLLOWER
I20260927 22:57:54.673964 27260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3802047216ae4349969d808c50459ba3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44025 } }
I20260927 22:57:54.674191 27260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:57:54.674338 27260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:57:54.674522 27260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:54.677047 27260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3802047216ae4349969d808c50459ba3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44025 } }
I20260927 22:57:54.677541 27260 leader_election.cc:302] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3802047216ae4349969d808c50459ba3; no voters:
I20260927 22:57:54.678164 27260 leader_election.cc:288] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:57:54.678323 27265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:54.678979 27265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 3802047216ae4349969d808c50459ba3, State: Running, Role: LEADER
I20260927 22:57:54.679549 27265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3802047216ae4349969d808c50459ba3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44025 } }
I20260927 22:57:54.680267 27260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:57:54.682353 27266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3802047216ae4349969d808c50459ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3802047216ae4349969d808c50459ba3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44025 } } }
I20260927 22:57:54.682376 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3802047216ae4349969d808c50459ba3. Latest consensus state: current_term: 1 leader_uuid: "3802047216ae4349969d808c50459ba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3802047216ae4349969d808c50459ba3" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44025 } } }
I20260927 22:57:54.682782 27266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:54.682782 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:54.685635 27271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:57:54.690505 27271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:57:54.699613 27271 catalog_manager.cc:1347] Generated new cluster ID: 6fa63e694de64de4ad699b5820b19e3c
I20260927 22:57:54.699744 27271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:57:54.710528 27271 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:57:54.712378 27271 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:57:54.732218 27271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3: Generated new TSK 0
I20260927 22:57:54.733155 27271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:57:54.798511 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44025
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:55.127794 27286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:55.128060 27286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:55.128154 27286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:55.135174 27286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:55.135325 27286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:55.135468 27286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:57:55.145147 27286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:55.146399 27286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:55.147701 27286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:55.160717 27291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:55.160717 27292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:55.160919 27294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:55.162621 27286 server_base.cc:1034] running on GCE node
I20260927 22:57:55.163206 27286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:55.164224 27286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:55.165469 27286 hybrid_clock.cc:648] HybridClock initialized: now 1790549875165432 us; error 47 us; skew 500 ppm
I20260927 22:57:55.165791 27286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:55 dist-test-slave-40vx krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:55.168965 27286 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:57:55.170360 27286 webserver.cc:460] Webserver started at http://127.8.204.129:46479/ using document root <none> and password file <none>
I20260927 22:57:55.171057 27286 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:55.171206 27286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:55.171546 27286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:55.174636 27286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "3fc0179261ad43e386505be6134a9aba"
format_stamp: "Formatted at 2026-09-27 22:57:55 on dist-test-slave-40vx"
server_key: "8a7e39597f01c49cb9145f9d117f88e5"
server_key_iv: "33486e0adc6248087b197d66dd519654"
server_key_version: "encryptionkey@0"
I20260927 22:57:55.175513 27286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "3fc0179261ad43e386505be6134a9aba"
format_stamp: "Formatted at 2026-09-27 22:57:55 on dist-test-slave-40vx"
server_key: "8a7e39597f01c49cb9145f9d117f88e5"
server_key_iv: "33486e0adc6248087b197d66dd519654"
server_key_version: "encryptionkey@0"
I20260927 22:57:55.180655 27286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 22:57:55.184326 27301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:55.185663 27286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:57:55.185853 27286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "3fc0179261ad43e386505be6134a9aba"
format_stamp: "Formatted at 2026-09-27 22:57:55 on dist-test-slave-40vx"
server_key: "8a7e39597f01c49cb9145f9d117f88e5"
server_key_iv: "33486e0adc6248087b197d66dd519654"
server_key_version: "encryptionkey@0"
I20260927 22:57:55.186066 27286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:55.208981 27286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:55.213248 27286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:55.213560 27286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:55.214574 27286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:55.215955 27286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:55.216068 27286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:55.216215 27286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:55.216295 27286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:55.249565 27286 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:37983
I20260927 22:57:55.249621 27413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:37983 every 8 connection(s)
I20260927 22:57:55.250874 27286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:57:55.252089 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27286
I20260927 22:57:55.252271 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:57:55.252818 9010 external_mini_cluster.cc:1428] Setting key a0541373552beeb6933e75b73b55a2cf
Sep 27 22:57:55 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:55.270761 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:55.253729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:42163 (local address 127.8.204.190:44025)
0927 22:57:55.254157 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:55.254163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:55.255313 (+ 1150us) server_negotiation.cc:407] Connection header received
0927 22:57:55.257143 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:55.257148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:55.257244 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:55.257360 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:55.258808 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.259585 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.261088 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.261494 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.261628 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:55.264983 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:55.264988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:55.264992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:55.265021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:55.267783 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:55.268494 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:55.268507 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:55.268509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:55.268621 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:55.269039 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:55.269044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:55.269047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:55.269316 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:55.269472 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:55.270147 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 22:57:55.270385 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":195,"threads_started":1}
I20260927 22:57:55.271785 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:55.254317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44025 (local address 127.8.204.129:42163)
0927 22:57:55.255130 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:55.255180 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:55.256892 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:55.257593 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:55.257601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:55.258096 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:55.258632 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:55.258649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.259738 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.259748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:55.260945 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:55.260957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.261722 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.261730 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:55.261959 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:55.262640 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:55.262661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:55.264769 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:55.267924 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:55.267932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:55.267948 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:55.268365 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:55.268731 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:55.268737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:55.268740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:55.268920 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:55.269440 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:55.269448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:55.269789 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 22:57:55.270050 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 22:57:55.270586 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":301,"threads_started":1}
I20260927 22:57:55.273341 27414 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44025
I20260927 22:57:55.273633 27414 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:55.274312 27414 heartbeater.cc:511] Master 127.8.204.190:44025 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:57:55.276559 27225 ts_manager.cc:194] Registered new tserver with Master: 3fc0179261ad43e386505be6134a9aba (127.8.204.129:37983)
I20260927 22:57:55.278002 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:42163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:57:55.307087 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44025
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:55.627972 27420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:55.628250 27420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:55.628413 27420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:55.635479 27420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:55.635613 27420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:55.635815 27420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:57:55.645419 27420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:55.646662 27420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:55.647972 27420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:55.660894 27425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:55.660953 27428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:55.660989 27426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:55.661863 27420 server_base.cc:1034] running on GCE node
I20260927 22:57:55.662529 27420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:55.663556 27420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:55.664757 27420 hybrid_clock.cc:648] HybridClock initialized: now 1790549875664723 us; error 64 us; skew 500 ppm
I20260927 22:57:55.665087 27420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:55 dist-test-slave-40vx krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:55.668299 27420 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:57:55.669773 27420 webserver.cc:460] Webserver started at http://127.8.204.130:39847/ using document root <none> and password file <none>
I20260927 22:57:55.670419 27420 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:55.670521 27420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:55.670780 27420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:55.673641 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "f2e48457dd1b4172a13107708dc61a0a"
format_stamp: "Formatted at 2026-09-27 22:57:55 on dist-test-slave-40vx"
server_key: "b5017ed92a7c83dce38da65ccf4f41d7"
server_key_iv: "f48112ac1222dec48705b939fe7b850a"
server_key_version: "encryptionkey@0"
I20260927 22:57:55.674381 27420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "f2e48457dd1b4172a13107708dc61a0a"
format_stamp: "Formatted at 2026-09-27 22:57:55 on dist-test-slave-40vx"
server_key: "b5017ed92a7c83dce38da65ccf4f41d7"
server_key_iv: "f48112ac1222dec48705b939fe7b850a"
server_key_version: "encryptionkey@0"
I20260927 22:57:55.679446 27420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:57:55.682948 27435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:55.684296 27420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:57:55.684451 27420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "f2e48457dd1b4172a13107708dc61a0a"
format_stamp: "Formatted at 2026-09-27 22:57:55 on dist-test-slave-40vx"
server_key: "b5017ed92a7c83dce38da65ccf4f41d7"
server_key_iv: "f48112ac1222dec48705b939fe7b850a"
server_key_version: "encryptionkey@0"
I20260927 22:57:55.684636 27420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:55.708384 27420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:55.712707 27420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:55.712997 27420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:55.714053 27420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:55.715317 27420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:55.715389 27420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:55.715505 27420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:55.715559 27420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:55.749207 27420 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38735
I20260927 22:57:55.749279 27547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38735 every 8 connection(s)
I20260927 22:57:55.750496 27420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:57:55.760416 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27420
I20260927 22:57:55.760592 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:57:55.761031 9010 external_mini_cluster.cc:1428] Setting key 9f2b54f30056a9f6c9a78c76e5656bfd
Sep 27 22:57:55 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549875, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:55.770337 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:55.753014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:44709 (local address 127.8.204.190:44025)
0927 22:57:55.753239 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:55.753245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:55.754362 (+ 1117us) server_negotiation.cc:407] Connection header received
0927 22:57:55.756387 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:55.756392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:55.756461 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:55.756551 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:55.757997 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.758682 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.760220 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.760777 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.760911 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:55.764533 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:55.764539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:55.764543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:55.764584 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:55.767393 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:55.768121 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:55.768126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:55.768130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:55.768258 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:55.768677 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:55.768682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:55.768685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:55.768947 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:55.769110 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:55.769763 (+ 653us) server_negotiation.cc:299] Negotiation successful
0927 22:57:55.769985 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:57:55.771245 27550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:55.753524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44025 (local address 127.8.204.130:44709)
0927 22:57:55.754202 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:55.754242 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:55.756127 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:55.756783 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:55.756791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:55.757287 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:55.757847 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:55.757864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.758801 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.758804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:55.760069 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:55.760081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:55.760989 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:55.760997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:55.761280 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:55.761984 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:55.762013 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:55.764326 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:55.767532 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:55.767540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:55.767554 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:55.767986 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:55.768369 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:55.768374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:55.768378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:55.768552 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:55.769082 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:55.769089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:55.769407 (+ 318us) client_negotiation.cc:769] Sending connection context
0927 22:57:55.769673 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 22:57:55.770169 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":266,"threads_started":1}
I20260927 22:57:55.772713 27548 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44025
I20260927 22:57:55.773010 27548 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:55.773674 27548 heartbeater.cc:511] Master 127.8.204.190:44025 requested a full tablet report, sending...
I20260927 22:57:55.775149 27225 ts_manager.cc:194] Registered new tserver with Master: f2e48457dd1b4172a13107708dc61a0a (127.8.204.130:38735)
I20260927 22:57:55.776448 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:44709
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:57:55.812726 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44025
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:34567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:57:56.136503 27553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:56.136775 27553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:56.136868 27553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:56.144027 27553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:56.144125 27553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:56.144235 27553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:57:56.153812 27553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:56.155037 27553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:56.156387 27553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:56.169605 27558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:56.169617 27561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:56.169617 27559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:56.170384 27553 server_base.cc:1034] running on GCE node
I20260927 22:57:56.170940 27553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:56.172011 27553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:56.173215 27553 hybrid_clock.cc:648] HybridClock initialized: now 1790549876173182 us; error 31 us; skew 500 ppm
I20260927 22:57:56.173529 27553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:57:56 dist-test-slave-40vx krb5kdc[27180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:57:56.176823 27553 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:57:56.178318 27553 webserver.cc:460] Webserver started at http://127.8.204.131:37699/ using document root <none> and password file <none>
I20260927 22:57:56.179009 27553 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:56.179144 27553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:56.179632 27553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:56.182745 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "2c45226081924a5898c20f1e1fc88127"
format_stamp: "Formatted at 2026-09-27 22:57:56 on dist-test-slave-40vx"
server_key: "8025807de851ea7d2c261a0fe7894bb8"
server_key_iv: "58c34f8437bc59da7ba320568f38c727"
server_key_version: "encryptionkey@0"
I20260927 22:57:56.183576 27553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "2c45226081924a5898c20f1e1fc88127"
format_stamp: "Formatted at 2026-09-27 22:57:56 on dist-test-slave-40vx"
server_key: "8025807de851ea7d2c261a0fe7894bb8"
server_key_iv: "58c34f8437bc59da7ba320568f38c727"
server_key_version: "encryptionkey@0"
I20260927 22:57:56.188851 27553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:57:56.192548 27568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:56.193886 27553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:57:56.194064 27553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "2c45226081924a5898c20f1e1fc88127"
format_stamp: "Formatted at 2026-09-27 22:57:56 on dist-test-slave-40vx"
server_key: "8025807de851ea7d2c261a0fe7894bb8"
server_key_iv: "58c34f8437bc59da7ba320568f38c727"
server_key_version: "encryptionkey@0"
I20260927 22:57:56.194274 27553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:56.208269 27553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:56.211997 27553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:56.212296 27553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:56.213191 27553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:56.214488 27553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:56.214588 27553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:56.214743 27553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:56.214813 27553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:56.247825 27553 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:33043
I20260927 22:57:56.247881 27680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:33043 every 8 connection(s)
I20260927 22:57:56.249253 27553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:57:56.255048 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27553
I20260927 22:57:56.255245 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:57:56.255683 9010 external_mini_cluster.cc:1428] Setting key aa0faa57c27bc057060c3025cda36192
Sep 27 22:57:56 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549876, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:57:56.270072 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.252179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:49605 (local address 127.8.204.190:44025)
0927 22:57:56.252403 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.252409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.253701 (+ 1292us) server_negotiation.cc:407] Connection header received
0927 22:57:56.256448 (+ 2747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.256453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.256575 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.256691 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:56.258149 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.258904 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.260458 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.260841 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.260972 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.264325 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:56.264330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:56.264333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:56.264364 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:56.267053 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.267841 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.267851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.267854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.267978 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.268403 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.268412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.268414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.268665 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:56.268821 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.269494 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.269714 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:57:56.271090 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.252719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:44025 (local address 127.8.204.131:49605)
0927 22:57:56.253495 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:56.253555 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:56.255856 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:56.256927 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:56.256936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:56.257445 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:56.257983 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.258000 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.259060 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.259065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.260307 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.260319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.261069 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.261075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.261303 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.261964 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:57:56.261985 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:56.264113 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:56.267199 (+ 3086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:56.267208 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:56.267224 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:56.267707 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:56.268090 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:57:56.268096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:57:56.268099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:57:56.268285 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:57:56.268799 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:56.268807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:56.269143 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 22:57:56.269399 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 22:57:56.269909 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":281,"threads_started":1}
I20260927 22:57:56.272532 27681 heartbeater.cc:347] Connected to a master server at 127.8.204.190:44025
I20260927 22:57:56.272801 27681 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:56.273450 27681 heartbeater.cc:511] Master 127.8.204.190:44025 requested a full tablet report, sending...
I20260927 22:57:56.274781 27225 ts_manager.cc:194] Registered new tserver with Master: 2c45226081924a5898c20f1e1fc88127 (127.8.204.131:33043)
I20260927 22:57:56.275663 27225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:49605
I20260927 22:57:56.281785 27414 heartbeater.cc:503] Master 127.8.204.190:44025 was elected leader, sending a full tablet report...
I20260927 22:57:56.282974 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:57:56.300246 27416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.286980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.8.204.190:44025)
0927 22:57:56.287142 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.287147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.287665 (+ 518us) server_negotiation.cc:407] Connection header received
0927 22:57:56.287791 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.287795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.287853 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.287935 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:56.289725 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.290484 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.291828 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.292230 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.292327 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.294899 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:56.294903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:56.294907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:56.294935 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:56.297341 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.298132 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.298135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.298138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.298260 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.298796 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.298803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.298811 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.299099 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:56.299398 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.299791 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.299967 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:57:56.304047 27224 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:57:56.310817 27224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:57:56.337349 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.323027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53808 (local address 127.8.204.129:37983)
0927 22:57:56.324159 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.324169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.325135 (+ 966us) server_negotiation.cc:407] Connection header received
0927 22:57:56.325546 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.325552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.325820 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.326014 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:56.327648 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.329761 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.333526 (+ 3765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.335489 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.335627 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.335970 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.336803 (+ 833us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.336980 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":827,"threads_started":1}
I20260927 22:57:56.337759 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.322684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:37983 (local address 127.0.0.1:53808)
0927 22:57:56.324979 (+ 2295us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:56.325024 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:56.325281 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:56.326732 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:56.326740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:56.326773 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:56.327394 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.327410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.330010 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.330018 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.333274 (+ 3256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.333295 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.336189 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.336198 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.336479 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.336555 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 22:57:56.336711 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 22:57:56.336978 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2010,"mutex_wait_us":4,"spinlock_wait_cycles":8064,"thread_start_us":1635,"threads_started":1}
I20260927 22:57:56.337806 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.323764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.8.204.130:38735)
0927 22:57:56.324617 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.324626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.325188 (+ 562us) server_negotiation.cc:407] Connection header received
0927 22:57:56.325513 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.325522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.325823 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.326048 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:56.328314 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.330417 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.335010 (+ 4593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.336476 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.336571 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.336736 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.337314 (+ 578us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.337505 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":522,"threads_started":1}
I20260927 22:57:56.338469 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.324486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38735 (local address 127.0.0.1:57850)
0927 22:57:56.325097 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:56.325114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:56.325281 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:56.326399 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:56.326410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:56.326463 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:56.328068 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.328085 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.332006 (+ 3921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.332014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.334812 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.334824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.336920 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.336928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.337103 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.337139 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:57:56.338067 (+ 928us) client_negotiation.cc:240] Negotiation successful
0927 22:57:56.338201 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":6,"spinlock_wait_cycles":6784,"thread_start_us":304,"threads_started":1}
I20260927 22:57:56.342639 27349 tablet_service.cc:1467] Processing CreateTablet for tablet e92c72cf6d9d4049a688d56269d1cef2 (DEFAULT_TABLE table=test-table [id=82092deca7cf4cafba149b69498c7c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:56.342639 27483 tablet_service.cc:1467] Processing CreateTablet for tablet e92c72cf6d9d4049a688d56269d1cef2 (DEFAULT_TABLE table=test-table [id=82092deca7cf4cafba149b69498c7c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:56.344165 27483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92c72cf6d9d4049a688d56269d1cef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:56.344156 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92c72cf6d9d4049a688d56269d1cef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:56.344871 27697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.325297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:33043 (local address 127.0.0.1:40174)
0927 22:57:56.334870 (+ 9573us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:56.334888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:56.334992 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:56.339057 (+ 4065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:56.339062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:56.339097 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:56.339445 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.339457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.341358 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.341365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.342601 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.342612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.343922 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.343929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.344099 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.344137 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:57:56.344390 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 22:57:56.344517 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9439,"thread_start_us":9786,"threads_started":1}
I20260927 22:57:56.344916 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.325840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.8.204.131:33043)
0927 22:57:56.338450 (+ 12610us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.338456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.338484 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:57:56.338586 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.338595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.338868 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.339056 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:56.339632 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.341210 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.342753 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.343733 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.343878 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.344107 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.344431 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.344573 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12448,"thread_start_us":12279,"threads_started":1}
I20260927 22:57:56.348330 27616 tablet_service.cc:1467] Processing CreateTablet for tablet e92c72cf6d9d4049a688d56269d1cef2 (DEFAULT_TABLE table=test-table [id=82092deca7cf4cafba149b69498c7c93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:56.349473 27616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e92c72cf6d9d4049a688d56269d1cef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:56.359004 27699 tablet_bootstrap.cc:492] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Bootstrap starting.
I20260927 22:57:56.359822 27700 tablet_bootstrap.cc:492] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Bootstrap starting.
I20260927 22:57:56.360855 27701 tablet_bootstrap.cc:492] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Bootstrap starting.
I20260927 22:57:56.364037 27701 tablet_bootstrap.cc:654] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:56.364656 27700 tablet_bootstrap.cc:654] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:56.364842 27699 tablet_bootstrap.cc:654] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:56.365139 27701 log.cc:824] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:56.365859 27700 log.cc:824] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:56.365917 27699 log.cc:824] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:56.367923 27701 tablet_bootstrap.cc:492] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: No bootstrap required, opened a new log
I20260927 22:57:56.368188 27701 ts_tablet_manager.cc:1397] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260927 22:57:56.373950 27700 tablet_bootstrap.cc:492] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: No bootstrap required, opened a new log
I20260927 22:57:56.374141 27699 tablet_bootstrap.cc:492] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: No bootstrap required, opened a new log
I20260927 22:57:56.374259 27700 ts_tablet_manager.cc:1397] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260927 22:57:56.374579 27699 ts_tablet_manager.cc:1397] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20260927 22:57:56.377532 27701 raft_consensus.cc:348] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.377992 27701 raft_consensus.cc:374] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:56.378103 27701 raft_consensus.cc:729] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c45226081924a5898c20f1e1fc88127, State: Initialized, Role: FOLLOWER
I20260927 22:57:56.378743 27701 consensus_queue.cc:260] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.379941 27681 heartbeater.cc:503] Master 127.8.204.190:44025 was elected leader, sending a full tablet report...
I20260927 22:57:56.380599 27701 ts_tablet_manager.cc:1428] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260927 22:57:56.384104 27699 raft_consensus.cc:348] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.384500 27699 raft_consensus.cc:374] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:56.384287 27700 raft_consensus.cc:348] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.384569 27699 raft_consensus.cc:729] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc0179261ad43e386505be6134a9aba, State: Initialized, Role: FOLLOWER
I20260927 22:57:56.384680 27700 raft_consensus.cc:374] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:56.384898 27700 raft_consensus.cc:729] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e48457dd1b4172a13107708dc61a0a, State: Initialized, Role: FOLLOWER
I20260927 22:57:56.385136 27699 consensus_queue.cc:260] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.385468 27700 consensus_queue.cc:260] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.386802 27699 ts_tablet_manager.cc:1428] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 22:57:56.387516 27548 heartbeater.cc:503] Master 127.8.204.190:44025 was elected leader, sending a full tablet report...
I20260927 22:57:56.388552 27700 ts_tablet_manager.cc:1428] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260927 22:57:56.402665 27705 raft_consensus.cc:482] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:57:56.402906 27705 raft_consensus.cc:504] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.404683 27705 leader_election.cc:288] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fc0179261ad43e386505be6134a9aba (127.8.204.129:37983), f2e48457dd1b4172a13107708dc61a0a (127.8.204.130:38735)
I20260927 22:57:56.411645 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.405098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:45549 (local address 127.8.204.129:37983)
0927 22:57:56.405272 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.405279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.405381 (+ 102us) server_negotiation.cc:407] Connection header received
0927 22:57:56.405615 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.405622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.405705 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.405778 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:56.406211 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.407257 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.409680 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.410841 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.410999 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.411069 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.411328 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.411443 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 22:57:56.411755 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.404919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:37983 (local address 127.8.204.131:45549)
0927 22:57:56.405268 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:56.405288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:56.405449 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:56.405829 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:56.405836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:56.405868 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:56.406085 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.406093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.407413 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.407417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.409532 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.409546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.410980 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.410983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.411106 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.411167 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 22:57:56.411290 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:57:56.411447 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":83}
I20260927 22:57:56.412045 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.405051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:59517 (local address 127.8.204.130:38735)
0927 22:57:56.405259 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.405285 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.405885 (+ 600us) server_negotiation.cc:407] Connection header received
0927 22:57:56.406063 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.406066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.406117 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.406218 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:56.406693 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.407640 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.409900 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.411071 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.411204 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.411306 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.411658 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.411809 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:57:56.412391 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.405015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38735 (local address 127.8.204.131:59517)
0927 22:57:56.405785 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:56.405801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:56.405946 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:56.406240 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:56.406243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:56.406261 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:56.406549 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.406556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.407778 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.407784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.409767 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:56.409777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.411306 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.411311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:56.411494 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.411523 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:57:56.411615 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 22:57:56.411994 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":2,"spinlock_wait_cycles":19200,"thread_start_us":441,"threads_started":1}
I20260927 22:57:56.412711 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92c72cf6d9d4049a688d56269d1cef2" candidate_uuid: "2c45226081924a5898c20f1e1fc88127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc0179261ad43e386505be6134a9aba" is_pre_election: true
I20260927 22:57:56.413216 27369 raft_consensus.cc:2454] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c45226081924a5898c20f1e1fc88127 in term 0.
I20260927 22:57:56.413283 27503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92c72cf6d9d4049a688d56269d1cef2" candidate_uuid: "2c45226081924a5898c20f1e1fc88127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e48457dd1b4172a13107708dc61a0a" is_pre_election: true
I20260927 22:57:56.413748 27503 raft_consensus.cc:2454] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c45226081924a5898c20f1e1fc88127 in term 0.
I20260927 22:57:56.413854 27572 leader_election.cc:302] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c45226081924a5898c20f1e1fc88127, 3fc0179261ad43e386505be6134a9aba; no voters:
I20260927 22:57:56.414217 27705 raft_consensus.cc:2781] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:57:56.414351 27705 raft_consensus.cc:482] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:57:56.414448 27705 raft_consensus.cc:3037] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:56.416711 27705 raft_consensus.cc:504] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.417323 27705 leader_election.cc:288] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [CANDIDATE]: Term 1 election: Requested vote from peers 3fc0179261ad43e386505be6134a9aba (127.8.204.129:37983), f2e48457dd1b4172a13107708dc61a0a (127.8.204.130:38735)
I20260927 22:57:56.417637 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92c72cf6d9d4049a688d56269d1cef2" candidate_uuid: "2c45226081924a5898c20f1e1fc88127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc0179261ad43e386505be6134a9aba"
I20260927 22:57:56.417768 27503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e92c72cf6d9d4049a688d56269d1cef2" candidate_uuid: "2c45226081924a5898c20f1e1fc88127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e48457dd1b4172a13107708dc61a0a"
I20260927 22:57:56.417842 27369 raft_consensus.cc:3037] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:56.417959 27503 raft_consensus.cc:3037] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:56.421038 27503 raft_consensus.cc:2454] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c45226081924a5898c20f1e1fc88127 in term 1.
I20260927 22:57:56.421038 27369 raft_consensus.cc:2454] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c45226081924a5898c20f1e1fc88127 in term 1.
I20260927 22:57:56.421449 27571 leader_election.cc:302] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c45226081924a5898c20f1e1fc88127, f2e48457dd1b4172a13107708dc61a0a; no voters:
I20260927 22:57:56.421742 27705 raft_consensus.cc:2781] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:56.422154 27705 raft_consensus.cc:686] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 1 LEADER]: Becoming Leader. State: Replica: 2c45226081924a5898c20f1e1fc88127, State: Running, Role: LEADER
I20260927 22:57:56.422572 27705 consensus_queue.cc:237] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } }
I20260927 22:57:56.427264 27225 catalog_manager.cc:5495] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c45226081924a5898c20f1e1fc88127 (127.8.204.131). New cstate: current_term: 1 leader_uuid: "2c45226081924a5898c20f1e1fc88127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c45226081924a5898c20f1e1fc88127" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 } health_report { overall_health: UNKNOWN } } }
I20260927 22:57:56.458554 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.453750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.8.204.131:33043)
0927 22:57:56.453934 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.453939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.453954 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:57:56.454024 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.454027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.454094 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.454214 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:57:56.454543 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.455312 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.456683 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.457063 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.457156 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.457588 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:57:56.458076 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:57:56.458197 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.458333 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.458368 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:57:56.466202 27503 raft_consensus.cc:1261] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 1 FOLLOWER]: Refusing update from remote peer 2c45226081924a5898c20f1e1fc88127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:56.466302 27369 raft_consensus.cc:1261] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 1 FOLLOWER]: Refusing update from remote peer 2c45226081924a5898c20f1e1fc88127: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:56.467206 27709 consensus_queue.cc:1035] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc0179261ad43e386505be6134a9aba" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 37983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:56.467581 27705 consensus_queue.cc:1035] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e48457dd1b4172a13107708dc61a0a" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:56.479641 27712 mvcc.cc:205] Tried to move back new op lower bound from 7334092293996191744 to 7334092293831516160. Current Snapshot: MvccSnapshot[applied={T|T < 7334092293996191744}]
I20260927 22:57:56.494524 27225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:56.494887 27225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:56.498430 27225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3802047216ae4349969d808c50459ba3: Sending DeleteTablet for 3 replicas of tablet e92c72cf6d9d4049a688d56269d1cef2
I20260927 22:57:56.499768 27483 tablet_service.cc:1514] Processing DeleteTablet for tablet e92c72cf6d9d4049a688d56269d1cef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:56 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:57850
I20260927 22:57:56.499768 27616 tablet_service.cc:1514] Processing DeleteTablet for tablet e92c72cf6d9d4049a688d56269d1cef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:56 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:40174
I20260927 22:57:56.500773 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet e92c72cf6d9d4049a688d56269d1cef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:56 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:53808
I20260927 22:57:56.502771 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27286
I20260927 22:57:56.511835 27723 tablet_replica.cc:330] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: stopping tablet replica
I20260927 22:57:56.512323 27723 raft_consensus.cc:2229] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:56.512893 27723 raft_consensus.cc:2258] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:56.515239 27723 ts_tablet_manager.cc:1905] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:56.517030 27721 tablet_replica.cc:330] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: stopping tablet replica
I20260927 22:57:56.517628 27721 raft_consensus.cc:2229] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:57:56.518460 27721 raft_consensus.cc:2258] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:56.520859 27721 ts_tablet_manager.cc:1905] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:56.522498 27723 ts_tablet_manager.cc:1918] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:56.522670 27723 log.cc:1196] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/e92c72cf6d9d4049a688d56269d1cef2
I20260927 22:57:56.523211 27723 ts_tablet_manager.cc:1939] T e92c72cf6d9d4049a688d56269d1cef2 P 3fc0179261ad43e386505be6134a9aba: Deleting consensus metadata
I20260927 22:57:56.523211 27722 tablet_replica.cc:330] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: stopping tablet replica
I20260927 22:57:56.523695 27722 raft_consensus.cc:2229] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:56.525130 27212 catalog_manager.cc:4841] TS 3fc0179261ad43e386505be6134a9aba (127.8.204.129:37983): tablet e92c72cf6d9d4049a688d56269d1cef2 (table test-table [id=82092deca7cf4cafba149b69498c7c93]) successfully deleted
I20260927 22:57:56.525707 27722 raft_consensus.cc:2258] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:57:56 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:57:56.527741 27722 ts_tablet_manager.cc:1905] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:56.530828 27721 ts_tablet_manager.cc:1918] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:56.531035 27721 log.cc:1196] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/e92c72cf6d9d4049a688d56269d1cef2
I20260927 22:57:56.531718 27721 ts_tablet_manager.cc:1939] T e92c72cf6d9d4049a688d56269d1cef2 P 2c45226081924a5898c20f1e1fc88127: Deleting consensus metadata
I20260927 22:57:56.533494 27212 catalog_manager.cc:4841] TS 2c45226081924a5898c20f1e1fc88127 (127.8.204.131:33043): tablet e92c72cf6d9d4049a688d56269d1cef2 (table test-table [id=82092deca7cf4cafba149b69498c7c93]) successfully deleted
I20260927 22:57:56.534320 27722 ts_tablet_manager.cc:1918] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:56.534451 27722 log.cc:1196] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/e92c72cf6d9d4049a688d56269d1cef2
I20260927 22:57:56.534744 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.505081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.8.204.129:37983)
0927 22:57:56.505258 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.505264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.515809 (+ 10545us) server_negotiation.cc:407] Connection header received
0927 22:57:56.516076 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.516083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.516172 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.516252 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:56.517928 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.519054 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.521055 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.521764 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.521854 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.527294 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:56.527309 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:56.527327 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:56.527375 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:56.531101 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.532170 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.532178 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.532182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.532331 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.532772 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.532778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.532781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.533229 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:56.533506 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.534048 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.534244 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:57:56.534950 27722 ts_tablet_manager.cc:1939] T e92c72cf6d9d4049a688d56269d1cef2 P f2e48457dd1b4172a13107708dc61a0a: Deleting consensus metadata
I20260927 22:57:56.535456 27409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:56.536623 27211 catalog_manager.cc:4841] TS f2e48457dd1b4172a13107708dc61a0a (127.8.204.130:38735): tablet e92c72cf6d9d4049a688d56269d1cef2 (table test-table [id=82092deca7cf4cafba149b69498c7c93]) successfully deleted
I20260927 22:57:56.625648 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27286
I20260927 22:57:56.649943 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27420
Sep 27 22:57:56 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:57:56.666987 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.650731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.8.204.130:38735)
0927 22:57:56.650906 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.650912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.650931 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:57:56.651143 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.651149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.651232 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.651373 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:56.653082 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.653981 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.655828 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.656204 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.656294 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.661286 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:56.661300 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:56.661316 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:56.661350 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:56.664266 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.664939 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.664946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.664948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.665066 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.665447 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.665453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.665456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.665819 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:56.666132 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.666430 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.666654 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:57:56.667708 27543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:56.758425 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27420
I20260927 22:57:56.775681 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27553
Sep 27 22:57:56 dist-test-slave-40vx krb5kdc[27180](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:57:56.793444 27698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:56.776560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.8.204.131:33043)
0927 22:57:56.776777 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:56.776783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:56.776807 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:57:56.776922 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:56.776927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:56.777021 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:56.777143 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:56.778372 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.779639 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.780942 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:56.781669 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:56.781802 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:56.785986 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:56.786001 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:56.786019 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:57:56.786065 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:56.790254 (+ 4189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.790951 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.790962 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.790967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.791173 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:57:56.791650 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:57:56.791656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:57:56.791662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:57:56.792147 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:56.792492 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:56.792735 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:57:56.792983 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:57:56.794294 27676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:56.916808 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27553
I20260927 22:57:56.935896 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27193
I20260927 22:57:56.937009 27255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:57.048784 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27193
2026-09-27T22:57:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4170 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:57:57Z Disabled control of system clock
I20260927 22:57:57.123210 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:41005
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:41005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:57:57.454546 27732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:57.454903 27732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:57.455070 27732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:57.462507 27732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:57:57.462613 27732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:57.462677 27732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:57.462766 27732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:57:57.462852 27732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:57:57.472208 27732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:41005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:41005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:57.473500 27732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:57.475196 27732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:57.486745 27738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:57.486709 27740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:57.486785 27737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:57.487566 27732 server_base.cc:1034] running on GCE node
I20260927 22:57:57.488402 27732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:57.489743 27732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:57.491007 27732 hybrid_clock.cc:648] HybridClock initialized: now 1790549877490957 us; error 27 us; skew 500 ppm
I20260927 22:57:57.491426 27732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:57.499111 27732 webserver.cc:460] Webserver started at http://127.8.204.190:32809/ using document root <none> and password file <none>
I20260927 22:57:57.499891 27732 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:57.500010 27732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:57.500339 27732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:57.503405 27732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "6796f502ec5146abb9955fc2ca0600a2"
format_stamp: "Formatted at 2026-09-27 22:57:57 on dist-test-slave-40vx"
server_key: "a8a1bcb7ece7a0e0e35bced64d2a8747"
server_key_iv: "167e2367ad0edf006dead461e2c208f3"
server_key_version: "encryptionkey@0"
I20260927 22:57:57.504209 27732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "6796f502ec5146abb9955fc2ca0600a2"
format_stamp: "Formatted at 2026-09-27 22:57:57 on dist-test-slave-40vx"
server_key: "a8a1bcb7ece7a0e0e35bced64d2a8747"
server_key_iv: "167e2367ad0edf006dead461e2c208f3"
server_key_version: "encryptionkey@0"
I20260927 22:57:57.509244 27732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:57:57.512603 27746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:57.513963 27732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:57:57.514149 27732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "6796f502ec5146abb9955fc2ca0600a2"
format_stamp: "Formatted at 2026-09-27 22:57:57 on dist-test-slave-40vx"
server_key: "a8a1bcb7ece7a0e0e35bced64d2a8747"
server_key_iv: "167e2367ad0edf006dead461e2c208f3"
server_key_version: "encryptionkey@0"
I20260927 22:57:57.514367 27732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:57.538772 27732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:57.539650 27732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:57.539951 27732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:57.558548 27732 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:41005
I20260927 22:57:57.558569 27797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:41005 every 8 connection(s)
I20260927 22:57:57.560004 27732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:57:57.563834 27798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:57.564106 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27732
I20260927 22:57:57.564284 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:57:57.564740 9010 external_mini_cluster.cc:1428] Setting key 828b969dc6cd8acac971e4fc6700ad6d
I20260927 22:57:57.576056 27798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2: Bootstrap starting.
I20260927 22:57:57.578477 27801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:57.566550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.8.204.190:41005)
0927 22:57:57.567660 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:57.567671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:57.567714 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:57:57.569918 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:57.569940 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:57.570421 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:57.570842 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:57.571418 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:57.572657 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:57.573995 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:57.574671 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:57.574867 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:57.576256 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:57.576264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:57.576285 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:57.576820 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:57.576895 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:57.576916 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:57.577045 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:57.577232 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 22:57:57.577558 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":354,"threads_started":1}
I20260927 22:57:57.579411 27798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:57.580448 27798 log.cc:824] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:57.583034 27798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2: No bootstrap required, opened a new log
I20260927 22:57:57.590070 27798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6796f502ec5146abb9955fc2ca0600a2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41005 } }
I20260927 22:57:57.590464 27798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:57.590600 27798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6796f502ec5146abb9955fc2ca0600a2, State: Initialized, Role: FOLLOWER
I20260927 22:57:57.591167 27798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6796f502ec5146abb9955fc2ca0600a2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41005 } }
I20260927 22:57:57.591476 27798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:57:57.591619 27798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:57:57.591804 27798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:57.593948 27798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6796f502ec5146abb9955fc2ca0600a2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41005 } }
I20260927 22:57:57.594444 27798 leader_election.cc:302] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6796f502ec5146abb9955fc2ca0600a2; no voters:
I20260927 22:57:57.595058 27798 leader_election.cc:288] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:57:57.595222 27803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:57.596005 27803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [term 1 LEADER]: Becoming Leader. State: Replica: 6796f502ec5146abb9955fc2ca0600a2, State: Running, Role: LEADER
I20260927 22:57:57.596560 27803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6796f502ec5146abb9955fc2ca0600a2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41005 } }
I20260927 22:57:57.597172 27798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:57:57.599581 27804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6796f502ec5146abb9955fc2ca0600a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6796f502ec5146abb9955fc2ca0600a2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41005 } } }
I20260927 22:57:57.599458 27805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6796f502ec5146abb9955fc2ca0600a2. Latest consensus state: current_term: 1 leader_uuid: "6796f502ec5146abb9955fc2ca0600a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6796f502ec5146abb9955fc2ca0600a2" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 41005 } } }
I20260927 22:57:57.599951 27805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:57.599956 27804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:57:57.600997 27809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:57:57.606601 27809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:57:57.615406 27809 catalog_manager.cc:1347] Generated new cluster ID: d9d71b220d4b43ed962e0cd65f4c3f12
I20260927 22:57:57.615514 27809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:57:57.642839 27809 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:57:57.644371 27809 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:57:57.656080 27809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2: Generated new TSK 0
I20260927 22:57:57.656973 27809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:57:57.677337 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41005
--builtin_ntp_servers=127.8.204.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:57:58.048041 27822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:58.048377 27822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:58.048539 27822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:58.055788 27822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:58.055929 27822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:58.056116 27822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:57:58.065377 27822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:58.066700 27822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:58.068526 27822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:58.081708 27827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:58.081920 27828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:58.082185 27830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:58.082636 27822 server_base.cc:1034] running on GCE node
I20260927 22:57:58.083205 27822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:58.084162 27822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:58.085397 27822 hybrid_clock.cc:648] HybridClock initialized: now 1790549878085349 us; error 57 us; skew 500 ppm
I20260927 22:57:58.085718 27822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:58.087983 27822 webserver.cc:460] Webserver started at http://127.8.204.129:46289/ using document root <none> and password file <none>
I20260927 22:57:58.088667 27822 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:58.088799 27822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:58.089161 27822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:58.091979 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "d29e3d4cc79942fea13e573581702cf3"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "71adc93e373fdff9cec75074f912d2ee"
server_key_iv: "edd081c17f5e5d5bed6241d63d5b190e"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.092746 27822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "d29e3d4cc79942fea13e573581702cf3"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "71adc93e373fdff9cec75074f912d2ee"
server_key_iv: "edd081c17f5e5d5bed6241d63d5b190e"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.097671 27822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:57:58.101408 27836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:58.102625 27822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:57:58.102808 27822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "d29e3d4cc79942fea13e573581702cf3"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "71adc93e373fdff9cec75074f912d2ee"
server_key_iv: "edd081c17f5e5d5bed6241d63d5b190e"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.103073 27822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:58.123961 27822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:58.124707 27822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:58.125000 27822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:58.125921 27822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:58.127331 27822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:58.127449 27822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:58.127580 27822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:58.127682 27822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:58.161121 27822 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:42757
I20260927 22:57:58.161221 27948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:42757 every 8 connection(s)
I20260927 22:57:58.162385 27822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:57:58.168133 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27822
I20260927 22:57:58.168326 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:57:58.168779 9010 external_mini_cluster.cc:1428] Setting key 5b87e3141d15f5d3e4ed7a5ed338f8c4
I20260927 22:57:58.171883 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41005
--builtin_ntp_servers=127.8.204.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:57:58.177718 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:58.164981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:39573 (local address 127.8.204.190:41005)
0927 22:57:58.165426 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:58.165432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:58.166668 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 22:57:58.169190 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:58.169196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:58.169286 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:58.169402 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:58.170859 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.172016 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.174663 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.175071 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.175197 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:58.176936 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:58.176941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:58.176945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:58.176964 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:58.177018 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:58.177031 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:58.177121 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:58.177373 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:57:58.177456 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":211,"threads_started":1}
I20260927 22:57:58.179258 27952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:58.165612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:41005 (local address 127.8.204.129:39573)
0927 22:57:58.166513 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:58.166553 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:58.168894 (+ 2341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:58.169659 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:58.169669 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:58.169811 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:58.170659 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:58.170686 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.172214 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.172223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:58.174488 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:58.174506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.175367 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.175376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:58.175720 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:58.176532 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:57:58.176537 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:58.176595 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:57:58.176720 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:58.177139 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:58.177152 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:58.177164 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:57:58.177362 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:57:58.177831 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":284,"threads_started":1}
I20260927 22:57:58.181005 27949 heartbeater.cc:347] Connected to a master server at 127.8.204.190:41005
I20260927 22:57:58.181360 27949 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:58.182368 27949 heartbeater.cc:511] Master 127.8.204.190:41005 requested a full tablet report, sending...
I20260927 22:57:58.184633 27763 ts_manager.cc:194] Registered new tserver with Master: d29e3d4cc79942fea13e573581702cf3 (127.8.204.129:42757)
I20260927 22:57:58.186129 27763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.129:39573
W20260927 22:57:58.495730 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:58.496022 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:58.496181 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:58.503695 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:58.503830 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:58.504060 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:57:58.513152 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:58.514354 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:58.516063 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:58.529181 27958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:58.529340 27959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:58.529313 27961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:58.530023 27953 server_base.cc:1034] running on GCE node
I20260927 22:57:58.530611 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:58.531589 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:58.532822 27953 hybrid_clock.cc:648] HybridClock initialized: now 1790549878532781 us; error 35 us; skew 500 ppm
I20260927 22:57:58.533141 27953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:58.535476 27953 webserver.cc:460] Webserver started at http://127.8.204.130:46797/ using document root <none> and password file <none>
I20260927 22:57:58.536159 27953 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:58.536284 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:58.536598 27953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:58.539428 27953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "0714fd29a5d4482e9d163abb7c7d3ddb"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "5ba5b097a87325b79d2ccda8b3f35edc"
server_key_iv: "abd6cdb1b4798ba3c56bd7cc34b67629"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.540191 27953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "0714fd29a5d4482e9d163abb7c7d3ddb"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "5ba5b097a87325b79d2ccda8b3f35edc"
server_key_iv: "abd6cdb1b4798ba3c56bd7cc34b67629"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.545136 27953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:57:58.548498 27967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:58.549711 27953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:57:58.549887 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "0714fd29a5d4482e9d163abb7c7d3ddb"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "5ba5b097a87325b79d2ccda8b3f35edc"
server_key_iv: "abd6cdb1b4798ba3c56bd7cc34b67629"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.550093 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:58.563055 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:58.563963 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:58.564261 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:58.565318 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:58.566952 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:58.567054 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:58.567157 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:58.567232 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:58.599876 27953 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:44135
I20260927 22:57:58.599954 28079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:44135 every 8 connection(s)
I20260927 22:57:58.601258 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:57:58.609202 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 27953
I20260927 22:57:58.609407 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:57:58.609920 9010 external_mini_cluster.cc:1428] Setting key 718f9abd82590f9db706e78299d974f6
I20260927 22:57:58.613040 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41005
--builtin_ntp_servers=127.8.204.148:41793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:57:58.615340 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:58.604067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:49571 (local address 127.8.204.190:41005)
0927 22:57:58.604270 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:58.604275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:58.605626 (+ 1351us) server_negotiation.cc:407] Connection header received
0927 22:57:58.607918 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:58.607922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:58.607989 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:58.608079 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:58.609368 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.610168 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.612458 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.612848 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.612940 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:58.614601 (+ 1661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:58.614605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:58.614608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:58.614625 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:58.614656 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:58.614667 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:58.614732 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:58.614997 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 22:57:58.615069 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:57:58.616902 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:58.604589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:41005 (local address 127.8.204.130:49571)
0927 22:57:58.605471 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:58.605513 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:58.607653 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:58.608298 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:58.608306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:58.608394 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:58.609136 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:58.609163 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.610326 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.610331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:58.612298 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:58.612314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:58.613078 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:58.613088 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:58.613455 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:58.614218 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:57:58.614225 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:58.614284 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:57:58.614393 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:58.614766 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:58.614780 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:58.614793 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:57:58.614984 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 22:57:58.615452 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260927 22:57:58.618333 28080 heartbeater.cc:347] Connected to a master server at 127.8.204.190:41005
I20260927 22:57:58.618655 28080 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:58.619645 28080 heartbeater.cc:511] Master 127.8.204.190:41005 requested a full tablet report, sending...
I20260927 22:57:58.621199 27763 ts_manager.cc:194] Registered new tserver with Master: 0714fd29a5d4482e9d163abb7c7d3ddb (127.8.204.130:44135)
I20260927 22:57:58.622025 27763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.130:49571
W20260927 22:57:58.951792 28083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:57:58.952045 28083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:57:58.952136 28083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:57:58.959311 28083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:57:58.959409 28083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:57:58.959511 28083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:57:58.968283 28083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:41005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:57:58.969424 28083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:57:58.971079 28083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:57:58.983767 28088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:58.983906 28089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:57:58.983963 28091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:57:58.985136 28083 server_base.cc:1034] running on GCE node
I20260927 22:57:58.985740 28083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:57:58.986673 28083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:57:58.987898 28083 hybrid_clock.cc:648] HybridClock initialized: now 1790549878987830 us; error 55 us; skew 500 ppm
I20260927 22:57:58.988245 28083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:57:58.990458 28083 webserver.cc:460] Webserver started at http://127.8.204.131:43755/ using document root <none> and password file <none>
I20260927 22:57:58.991132 28083 fs_manager.cc:359] Metadata directory not provided
I20260927 22:57:58.991243 28083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:57:58.991590 28083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:57:58.994604 28083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "b207b8c0b8f5c4071beff0bb50f5fe7b"
server_key_iv: "bf0de598c98e2462b37d16febca112d7"
server_key_version: "encryptionkey@0"
I20260927 22:57:58.995496 28083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "b207b8c0b8f5c4071beff0bb50f5fe7b"
server_key_iv: "bf0de598c98e2462b37d16febca112d7"
server_key_version: "encryptionkey@0"
I20260927 22:57:59.000640 28083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:57:59.004101 28097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:59.005499 28083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:57:59.005661 28083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c"
format_stamp: "Formatted at 2026-09-27 22:57:58 on dist-test-slave-40vx"
server_key: "b207b8c0b8f5c4071beff0bb50f5fe7b"
server_key_iv: "bf0de598c98e2462b37d16febca112d7"
server_key_version: "encryptionkey@0"
I20260927 22:57:59.005867 28083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:57:59.050668 28083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:57:59.051600 28083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:57:59.051901 28083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:57:59.052860 28083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:57:59.054430 28083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:57:59.054517 28083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:59.054594 28083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:57:59.054673 28083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:57:59.087517 28083 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:41889
I20260927 22:57:59.087565 28209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:41889 every 8 connection(s)
I20260927 22:57:59.088802 28083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:57:59.089859 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28083
I20260927 22:57:59.090034 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:57:59.090571 9010 external_mini_cluster.cc:1428] Setting key 982d92ea92dfee2d31c5da917adfd451
I20260927 22:57:59.101429 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.091587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38457 (local address 127.8.204.190:41005)
0927 22:57:59.091811 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.091818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.093497 (+ 1679us) server_negotiation.cc:407] Connection header received
0927 22:57:59.095232 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.095237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.095337 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.095454 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:59.096453 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.097186 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.098794 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.099217 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.099395 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.100559 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:59.100565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:59.100569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:59.100597 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:59.100626 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:59.100638 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:59.100757 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.101008 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.101100 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:57:59.102411 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.092266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:41005 (local address 127.8.204.131:38457)
0927 22:57:59.093349 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:59.093408 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:59.094978 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:59.095676 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:59.095684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:59.095776 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:57:59.096300 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.096315 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.097351 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.097356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.098626 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.098637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.099483 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.099491 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.099709 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.100280 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:57:59.100283 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:57:59.100318 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:57:59.100384 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:57:59.100787 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:57:59.100803 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:57:59.100809 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:57:59.100992 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 22:57:59.101335 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":348,"threads_started":1}
I20260927 22:57:59.103722 28210 heartbeater.cc:347] Connected to a master server at 127.8.204.190:41005
I20260927 22:57:59.104012 28210 heartbeater.cc:464] Registering TS with master...
I20260927 22:57:59.104794 28210 heartbeater.cc:511] Master 127.8.204.190:41005 requested a full tablet report, sending...
I20260927 22:57:59.106127 27763 ts_manager.cc:194] Registered new tserver with Master: a1938b7eaf61489a9e4c5fc5fd182e8c (127.8.204.131:41889)
I20260927 22:57:59.107017 27763 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.131:38457
I20260927 22:57:59.117836 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:57:59.127498 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.121916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.8.204.190:41005)
0927 22:57:59.122074 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.122078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.122512 (+ 434us) server_negotiation.cc:407] Connection header received
0927 22:57:59.122637 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.122640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.122735 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.122819 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:59.123257 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.124005 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.125408 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.125811 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.125901 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.126861 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:59.126866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:59.126869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:59.126885 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:59.126913 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:59.126924 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:59.127029 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.127178 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.127222 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:57:59.130929 27763 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:57:59.138309 27763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:57:59.170770 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.150450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.8.204.130:44135)
0927 22:57:59.161308 (+ 10858us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.161322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.162286 (+ 964us) server_negotiation.cc:407] Connection header received
0927 22:57:59.162528 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.162536 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.162741 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.162903 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:59.163813 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.165329 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.168353 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.169614 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.169778 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.170029 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.170274 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.170433 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10616,"thread_start_us":10862,"threads_started":1}
I20260927 22:57:59.170795 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.149919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:44135 (local address 127.0.0.1:56122)
0927 22:57:59.162140 (+ 12221us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:59.162161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:59.162317 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:59.163210 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:59.163214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:59.163240 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:59.163632 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.163643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.165492 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.165498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.168123 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.168140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.169807 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.169813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.170017 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.170074 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 22:57:59.170211 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 22:57:59.170401 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12076,"mutex_wait_us":129,"thread_start_us":780,"threads_started":1}
I20260927 22:57:59.171649 28223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.149776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:41889 (local address 127.0.0.1:54080)
0927 22:57:59.150782 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:59.150838 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:59.151047 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:59.162655 (+ 11608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:59.162665 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:59.162711 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:59.163091 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.163105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.165407 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.165415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.168362 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.168377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.170608 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.170617 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.170855 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.170908 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:57:59.170977 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 22:57:59.171151 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":86,"thread_start_us":12548,"threads_started":1}
I20260927 22:57:59.173017 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.150088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.8.204.131:41889)
0927 22:57:59.161990 (+ 11902us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.161996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.162024 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:57:59.162103 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.162108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.162375 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.163081 (+ 706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:59.163382 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.165237 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.168588 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.170363 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.171944 (+ 1581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.172283 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.172360 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.172558 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11759,"thread_start_us":11618,"threads_started":1}
I20260927 22:57:59.177138 28145 tablet_service.cc:1467] Processing CreateTablet for tablet f53cbab011ea4050add21be00e9d9894 (DEFAULT_TABLE table=test-table [id=6ae7982bd8e748e4b2e5308d6d339cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:59.178615 28145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53cbab011ea4050add21be00e9d9894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:59.178576 28015 tablet_service.cc:1467] Processing CreateTablet for tablet f53cbab011ea4050add21be00e9d9894 (DEFAULT_TABLE table=test-table [id=6ae7982bd8e748e4b2e5308d6d339cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:59.179914 28015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53cbab011ea4050add21be00e9d9894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:59.181197 28222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.151658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42757 (local address 127.0.0.1:51672)
0927 22:57:59.171376 (+ 19718us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:59.171392 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:59.171467 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:59.174702 (+ 3235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:59.174706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:59.174733 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:59.174899 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.174909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.176793 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.176797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.178607 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.178622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.180425 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.180431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.180606 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.180651 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:57:59.180757 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:57:59.180912 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19578,"thread_start_us":23462,"threads_started":1}
I20260927 22:57:59.181339 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.152206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.8.204.129:42757)
0927 22:57:59.173986 (+ 21780us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.173997 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.174036 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:57:59.174154 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.174165 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.174474 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.174706 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:59.175104 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.176632 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.178804 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.180252 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.180397 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.180652 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.180805 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.180973 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21583,"thread_start_us":21469,"threads_started":1}
I20260927 22:57:59.184787 27884 tablet_service.cc:1467] Processing CreateTablet for tablet f53cbab011ea4050add21be00e9d9894 (DEFAULT_TABLE table=test-table [id=6ae7982bd8e748e4b2e5308d6d339cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:57:59.186094 27884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53cbab011ea4050add21be00e9d9894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:57:59.189724 27949 heartbeater.cc:503] Master 127.8.204.190:41005 was elected leader, sending a full tablet report...
I20260927 22:57:59.190076 28228 tablet_bootstrap.cc:492] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Bootstrap starting.
I20260927 22:57:59.194329 28228 tablet_bootstrap.cc:654] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:59.194518 28229 tablet_bootstrap.cc:492] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Bootstrap starting.
I20260927 22:57:59.195847 28228 log.cc:824] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:59.198076 28229 tablet_bootstrap.cc:654] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:59.199174 28228 tablet_bootstrap.cc:492] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: No bootstrap required, opened a new log
I20260927 22:57:59.199312 28229 log.cc:824] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:59.199482 28228 ts_tablet_manager.cc:1397] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:57:59.199944 28231 tablet_bootstrap.cc:492] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Bootstrap starting.
I20260927 22:57:59.202523 28229 tablet_bootstrap.cc:492] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: No bootstrap required, opened a new log
I20260927 22:57:59.202775 28229 ts_tablet_manager.cc:1397] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 22:57:59.203208 28231 tablet_bootstrap.cc:654] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Neither blocks nor log segments found. Creating new log.
I20260927 22:57:59.204272 28231 log.cc:824] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:57:59.206889 28231 tablet_bootstrap.cc:492] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: No bootstrap required, opened a new log
I20260927 22:57:59.207140 28231 ts_tablet_manager.cc:1397] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260927 22:57:59.210753 28228 raft_consensus.cc:348] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.211191 28228 raft_consensus.cc:374] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:59.211339 28228 raft_consensus.cc:729] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1938b7eaf61489a9e4c5fc5fd182e8c, State: Initialized, Role: FOLLOWER
I20260927 22:57:59.211969 28228 consensus_queue.cc:260] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.213075 28210 heartbeater.cc:503] Master 127.8.204.190:41005 was elected leader, sending a full tablet report...
I20260927 22:57:59.213686 28228 ts_tablet_manager.cc:1428] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260927 22:57:59.214272 28229 raft_consensus.cc:348] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.214715 28229 raft_consensus.cc:374] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:59.214830 28229 raft_consensus.cc:729] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0714fd29a5d4482e9d163abb7c7d3ddb, State: Initialized, Role: FOLLOWER
I20260927 22:57:59.214956 28231 raft_consensus.cc:348] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.215492 28231 raft_consensus.cc:374] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:57:59.215619 28231 raft_consensus.cc:729] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d29e3d4cc79942fea13e573581702cf3, State: Initialized, Role: FOLLOWER
I20260927 22:57:59.215399 28229 consensus_queue.cc:260] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.216235 28231 consensus_queue.cc:260] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.217033 28080 heartbeater.cc:503] Master 127.8.204.190:41005 was elected leader, sending a full tablet report...
I20260927 22:57:59.217543 28229 ts_tablet_manager.cc:1428] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 22:57:59.218880 28231 ts_tablet_manager.cc:1428] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
W20260927 22:57:59.340655 28211 tablet.cc:2369] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:57:59.353467 28081 tablet.cc:2369] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:57:59.375602 28234 raft_consensus.cc:482] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:57:59.375922 28234 raft_consensus.cc:504] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.377521 28234 leader_election.cc:288] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0714fd29a5d4482e9d163abb7c7d3ddb (127.8.204.130:44135), d29e3d4cc79942fea13e573581702cf3 (127.8.204.129:42757)
I20260927 22:57:59.383368 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.378051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:41309 (local address 127.8.204.130:44135)
0927 22:57:59.378266 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.378271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.378302 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:57:59.378547 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.378555 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.378656 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.378790 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:59.379324 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.380045 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.381692 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.382460 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.382618 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.382688 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.382946 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.383074 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:57:59.383371 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.377953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:44135 (local address 127.8.204.131:41309)
0927 22:57:59.378188 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:59.378211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:59.378349 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:59.378968 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:59.378971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:59.378993 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:59.379155 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.379162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.380176 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.380184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.381543 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.381558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.382609 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.382614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.382779 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.382802 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 22:57:59.382920 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:57:59.383054 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":44}
I20260927 22:57:59.384198 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.378679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:38519 (local address 127.8.204.129:42757)
0927 22:57:59.378895 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.378901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.379200 (+ 299us) server_negotiation.cc:407] Connection header received
0927 22:57:59.379473 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.379478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.379561 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.379660 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:57:59.380192 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.380866 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.382533 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.383357 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.383513 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.383578 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.383854 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.383981 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:57:59.384268 28237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.378410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:42757 (local address 127.8.204.131:38519)
0927 22:57:59.379082 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:57:59.379099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:57:59.379244 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:57:59.379673 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:57:59.379678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:57:59.379700 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:57:59.380026 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.380035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.381007 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.381012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.382342 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:57:59.382366 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.383514 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.383520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:57:59.383696 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.383724 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:57:59.383825 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:57:59.383937 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":1,"thread_start_us":370,"threads_started":1}
I20260927 22:57:59.384511 28035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53cbab011ea4050add21be00e9d9894" candidate_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" is_pre_election: true
I20260927 22:57:59.385026 28035 raft_consensus.cc:2454] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1938b7eaf61489a9e4c5fc5fd182e8c in term 0.
I20260927 22:57:59.385236 27904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53cbab011ea4050add21be00e9d9894" candidate_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29e3d4cc79942fea13e573581702cf3" is_pre_election: true
I20260927 22:57:59.385679 28100 leader_election.cc:302] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0714fd29a5d4482e9d163abb7c7d3ddb, a1938b7eaf61489a9e4c5fc5fd182e8c; no voters:
I20260927 22:57:59.385699 27904 raft_consensus.cc:2454] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1938b7eaf61489a9e4c5fc5fd182e8c in term 0.
I20260927 22:57:59.386046 28234 raft_consensus.cc:2781] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:57:59.386209 28234 raft_consensus.cc:482] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:57:59.386348 28234 raft_consensus.cc:3037] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:59.388607 28234 raft_consensus.cc:504] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.389245 28234 leader_election.cc:288] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [CANDIDATE]: Term 1 election: Requested vote from peers 0714fd29a5d4482e9d163abb7c7d3ddb (127.8.204.130:44135), d29e3d4cc79942fea13e573581702cf3 (127.8.204.129:42757)
I20260927 22:57:59.389642 28035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53cbab011ea4050add21be00e9d9894" candidate_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb"
I20260927 22:57:59.389746 27904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53cbab011ea4050add21be00e9d9894" candidate_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d29e3d4cc79942fea13e573581702cf3"
I20260927 22:57:59.389871 28035 raft_consensus.cc:3037] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:59.389947 27904 raft_consensus.cc:3037] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:57:59.392122 28035 raft_consensus.cc:2454] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1938b7eaf61489a9e4c5fc5fd182e8c in term 1.
I20260927 22:57:59.392187 27904 raft_consensus.cc:2454] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1938b7eaf61489a9e4c5fc5fd182e8c in term 1.
I20260927 22:57:59.392552 28100 leader_election.cc:302] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0714fd29a5d4482e9d163abb7c7d3ddb, a1938b7eaf61489a9e4c5fc5fd182e8c; no voters:
I20260927 22:57:59.392860 28234 raft_consensus.cc:2781] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:57:59.393285 28234 raft_consensus.cc:686] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 1 LEADER]: Becoming Leader. State: Replica: a1938b7eaf61489a9e4c5fc5fd182e8c, State: Running, Role: LEADER
I20260927 22:57:59.393733 28234 consensus_queue.cc:237] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } }
I20260927 22:57:59.398125 27762 catalog_manager.cc:5495] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c reported cstate change: term changed from 0 to 1, leader changed from <none> to a1938b7eaf61489a9e4c5fc5fd182e8c (127.8.204.131). New cstate: current_term: 1 leader_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1938b7eaf61489a9e4c5fc5fd182e8c" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 41889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 } health_report { overall_health: UNKNOWN } } }
W20260927 22:57:59.414762 27950 tablet.cc:2369] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:57:59.458861 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.453582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.8.204.131:41889)
0927 22:57:59.453823 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.453829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.453847 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:57:59.453929 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.453933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.454006 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.454157 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:57:59.454557 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.455335 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.456640 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.457069 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.457216 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.457775 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:57:59.458270 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:57:59.458454 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.458591 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.458635 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:57:59.467492 28035 raft_consensus.cc:1261] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 1 FOLLOWER]: Refusing update from remote peer a1938b7eaf61489a9e4c5fc5fd182e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:59.467492 27904 raft_consensus.cc:1261] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 1 FOLLOWER]: Refusing update from remote peer a1938b7eaf61489a9e4c5fc5fd182e8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:57:59.468474 28234 consensus_queue.cc:1035] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0714fd29a5d4482e9d163abb7c7d3ddb" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 44135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:59.468770 28238 consensus_queue.cc:1035] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d29e3d4cc79942fea13e573581702cf3" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 42757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:57:59.480712 28241 mvcc.cc:205] Tried to move back new op lower bound from 7334092306287210496 to 7334092306000154624. Current Snapshot: MvccSnapshot[applied={T|T < 7334092306287210496}]
I20260927 22:57:59.495121 27762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:59.495453 27762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:57:59.498872 27762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6796f502ec5146abb9955fc2ca0600a2: Sending DeleteTablet for 3 replicas of tablet f53cbab011ea4050add21be00e9d9894
I20260927 22:57:59.500095 28145 tablet_service.cc:1514] Processing DeleteTablet for tablet f53cbab011ea4050add21be00e9d9894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:59 UTC) from {username='slave'} at 127.0.0.1:54080
I20260927 22:57:59.500222 28015 tablet_service.cc:1514] Processing DeleteTablet for tablet f53cbab011ea4050add21be00e9d9894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:59 UTC) from {username='slave'} at 127.0.0.1:56122
I20260927 22:57:59.500362 27884 tablet_service.cc:1514] Processing DeleteTablet for tablet f53cbab011ea4050add21be00e9d9894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:57:59 UTC) from {username='slave'} at 127.0.0.1:51672
I20260927 22:57:59.502194 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27822
I20260927 22:57:59.512702 28251 tablet_replica.cc:330] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: stopping tablet replica
I20260927 22:57:59.513358 28252 tablet_replica.cc:330] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: stopping tablet replica
I20260927 22:57:59.514118 28252 raft_consensus.cc:2229] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:59.514737 28251 raft_consensus.cc:2229] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:57:59.515446 28251 raft_consensus.cc:2258] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:59.516050 28252 raft_consensus.cc:2258] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:59.520197 28251 ts_tablet_manager.cc:1905] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:59.522682 28252 ts_tablet_manager.cc:1905] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:59.524051 28250 tablet_replica.cc:330] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: stopping tablet replica
I20260927 22:57:59.524480 28250 raft_consensus.cc:2229] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:57:59.525234 28250 raft_consensus.cc:2258] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:57:59.529397 28250 ts_tablet_manager.cc:1905] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:57:59.529650 28252 ts_tablet_manager.cc:1918] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:59.529820 28252 log.cc:1196] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/f53cbab011ea4050add21be00e9d9894
I20260927 22:57:59.530342 28252 ts_tablet_manager.cc:1939] T f53cbab011ea4050add21be00e9d9894 P d29e3d4cc79942fea13e573581702cf3: Deleting consensus metadata
I20260927 22:57:59.530645 28251 ts_tablet_manager.cc:1918] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:59.530866 28251 log.cc:1196] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/f53cbab011ea4050add21be00e9d9894
I20260927 22:57:59.531478 28251 ts_tablet_manager.cc:1939] T f53cbab011ea4050add21be00e9d9894 P 0714fd29a5d4482e9d163abb7c7d3ddb: Deleting consensus metadata
I20260927 22:57:59.532199 27749 catalog_manager.cc:4841] TS d29e3d4cc79942fea13e573581702cf3 (127.8.204.129:42757): tablet f53cbab011ea4050add21be00e9d9894 (table test-table [id=6ae7982bd8e748e4b2e5308d6d339cf7]) successfully deleted
I20260927 22:57:59.533432 27749 catalog_manager.cc:4841] TS 0714fd29a5d4482e9d163abb7c7d3ddb (127.8.204.130:44135): tablet f53cbab011ea4050add21be00e9d9894 (table test-table [id=6ae7982bd8e748e4b2e5308d6d339cf7]) successfully deleted
I20260927 22:57:59.534384 28227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.504062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48336 (local address 127.8.204.129:42757)
0927 22:57:59.504305 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.504314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.519299 (+ 14985us) server_negotiation.cc:407] Connection header received
0927 22:57:59.519758 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.519764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.519868 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.519947 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:59.520603 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.521677 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.525601 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.526311 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.526410 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.532647 (+ 6237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:59.532658 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:59.532675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:59.533410 (+ 735us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:59.533473 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:59.533512 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:59.533629 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.533901 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.534019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:57:59.535043 27944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:59.538586 28250 ts_tablet_manager.cc:1918] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:57:59.538854 28250 log.cc:1196] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/f53cbab011ea4050add21be00e9d9894
I20260927 22:57:59.539701 28250 ts_tablet_manager.cc:1939] T f53cbab011ea4050add21be00e9d9894 P a1938b7eaf61489a9e4c5fc5fd182e8c: Deleting consensus metadata
I20260927 22:57:59.541826 27748 catalog_manager.cc:4841] TS a1938b7eaf61489a9e4c5fc5fd182e8c (127.8.204.131:41889): tablet f53cbab011ea4050add21be00e9d9894 (table test-table [id=6ae7982bd8e748e4b2e5308d6d339cf7]) successfully deleted
I20260927 22:57:59.638374 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27822
I20260927 22:57:59.656499 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27953
I20260927 22:57:59.664201 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.657324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.8.204.130:44135)
0927 22:57:59.657517 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.657524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.657560 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:57:59.657692 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.657706 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.657817 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.657928 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:59.658276 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.659453 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.660719 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.661382 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.661504 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.662529 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:59.662540 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:59.662560 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:59.663405 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:59.663467 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:59.663492 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:59.663604 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.663724 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.663833 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:57:59.664892 28075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:59.757170 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27953
I20260927 22:57:59.774328 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28083
I20260927 22:57:59.782477 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:57:59.775155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.8.204.131:41889)
0927 22:57:59.775405 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:57:59.775410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:57:59.775466 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:57:59.775669 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:57:59.775673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:57:59.775780 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:57:59.775860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:57:59.776352 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.777143 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.779192 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:57:59.779737 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:57:59.779836 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:57:59.781020 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:57:59.781028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:57:59.781046 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:57:59.781659 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:57:59.781706 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:57:59.781728 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:57:59.781820 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:57:59.781996 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 22:57:59.782079 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:57:59.783237 28205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:57:59.908357 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28083
I20260927 22:57:59.927940 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 27732
I20260927 22:57:59.929087 27793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:00.039062 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 27732
2026-09-27T22:58:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:00 dist-test-slave-40vx krb5kdc[28260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:00 dist-test-slave-40vx krb5kdc[28260](info): set up 2 sockets
Sep 27 22:58:00 dist-test-slave-40vx krb5kdc[28260](info): commencing operation
krb5kdc: starting...
W20260927 22:58:01.155648 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:01 dist-test-slave-40vx krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:01.403892 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:46333
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:01.728489 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:01.728857 28273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:01.729019 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:01.736402 28273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:01.736609 28273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:01.736732 28273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:01.736840 28273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:01.736927 28273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:01.746733 28273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:46333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:01.748128 28273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:01.749424 28273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:01.762264 28278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:01.762248 28281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:01.762418 28279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:01.763041 28273 server_base.cc:1034] running on GCE node
I20260927 22:58:01.763876 28273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:01.765244 28273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:01.766441 28273 hybrid_clock.cc:648] HybridClock initialized: now 1790549881766390 us; error 41 us; skew 500 ppm
I20260927 22:58:01.766793 28273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:01 dist-test-slave-40vx krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549881, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:01.769798 28273 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:01.771202 28273 webserver.cc:460] Webserver started at http://127.8.204.190:40059/ using document root <none> and password file <none>
I20260927 22:58:01.771916 28273 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:01.772043 28273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:01.772354 28273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:01.775182 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "d51e7f249e16449785c060cb63b64ab8"
format_stamp: "Formatted at 2026-09-27 22:58:01 on dist-test-slave-40vx"
server_key: "ae66a2a58deb5d98dce4869c962a6f3c"
server_key_iv: "af96bb34bbadb5879816f45b1cb3f54d"
server_key_version: "encryptionkey@0"
I20260927 22:58:01.776031 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "d51e7f249e16449785c060cb63b64ab8"
format_stamp: "Formatted at 2026-09-27 22:58:01 on dist-test-slave-40vx"
server_key: "ae66a2a58deb5d98dce4869c962a6f3c"
server_key_iv: "af96bb34bbadb5879816f45b1cb3f54d"
server_key_version: "encryptionkey@0"
I20260927 22:58:01.781036 28273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:58:01.784390 28288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:01.785701 28273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:01.785866 28273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "d51e7f249e16449785c060cb63b64ab8"
format_stamp: "Formatted at 2026-09-27 22:58:01 on dist-test-slave-40vx"
server_key: "ae66a2a58deb5d98dce4869c962a6f3c"
server_key_iv: "af96bb34bbadb5879816f45b1cb3f54d"
server_key_version: "encryptionkey@0"
I20260927 22:58:01.786046 28273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:01.813632 28273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:01.818333 28273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:01.818572 28273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:01.836472 28273 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:46333
I20260927 22:58:01.836490 28339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:46333 every 8 connection(s)
I20260927 22:58:01.837846 28273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:01.841519 28340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:01.847530 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28273
I20260927 22:58:01.847721 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:01.848165 9010 external_mini_cluster.cc:1428] Setting key 844c888fa7c177b2f6ceacb6bc004516
I20260927 22:58:01.852010 28340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8: Bootstrap starting.
I20260927 22:58:01.855742 28340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:01.856838 28340 log.cc:824] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:01.859825 28340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8: No bootstrap required, opened a new log
Sep 27 22:58:01 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:01.867877 28340 raft_consensus.cc:348] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e7f249e16449785c060cb63b64ab8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46333 } }
I20260927 22:58:01.868227 28340 raft_consensus.cc:374] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:01.868340 28340 raft_consensus.cc:729] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51e7f249e16449785c060cb63b64ab8, State: Initialized, Role: FOLLOWER
I20260927 22:58:01.868994 28340 consensus_queue.cc:260] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e7f249e16449785c060cb63b64ab8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46333 } }
I20260927 22:58:01.869267 28340 raft_consensus.cc:388] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:01.869416 28340 raft_consensus.cc:482] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:01.869607 28340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:01.870607 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:01.850005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.8.204.190:46333)
0927 22:58:01.850762 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:01.850778 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:01.850817 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:58:01.852549 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:01.852570 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:01.852937 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:01.853245 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:01.855098 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:01.856495 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:01.858113 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:01.858748 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:01.858879 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:01.863396 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:01.863404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:01.863417 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:01.863453 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:01.866489 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:01.867175 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:01.867185 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:01.867197 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:01.867413 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:01.867788 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:01.867796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:01.867800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:01.868406 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:01.868701 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:01.869148 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 22:58:01.869592 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":338,"threads_started":1}
I20260927 22:58:01.872148 28340 raft_consensus.cc:504] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e7f249e16449785c060cb63b64ab8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46333 } }
I20260927 22:58:01.872716 28340 leader_election.cc:302] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d51e7f249e16449785c060cb63b64ab8; no voters:
I20260927 22:58:01.873473 28340 leader_election.cc:288] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:01.873500 28345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:01.874257 28345 raft_consensus.cc:686] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [term 1 LEADER]: Becoming Leader. State: Replica: d51e7f249e16449785c060cb63b64ab8, State: Running, Role: LEADER
I20260927 22:58:01.874773 28345 consensus_queue.cc:237] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e7f249e16449785c060cb63b64ab8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46333 } }
I20260927 22:58:01.875722 28340 sys_catalog.cc:564] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:01.877787 28347 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d51e7f249e16449785c060cb63b64ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e7f249e16449785c060cb63b64ab8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46333 } } }
I20260927 22:58:01.878161 28347 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:01.878683 28346 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51e7f249e16449785c060cb63b64ab8. Latest consensus state: current_term: 1 leader_uuid: "d51e7f249e16449785c060cb63b64ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51e7f249e16449785c060cb63b64ab8" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46333 } } }
I20260927 22:58:01.878991 28346 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:01.881549 28353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:01.886518 28353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:01.895432 28353 catalog_manager.cc:1347] Generated new cluster ID: 897e02784b804553a41fbda1908b84e9
I20260927 22:58:01.895550 28353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:01.920996 28353 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:01.922533 28353 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:01.931226 28353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8: Generated new TSK 0
I20260927 22:58:01.932194 28353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.204.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
Entry for principal kudu/127.8.204.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab.
I20260927 22:58:01.992888 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:46333
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:02.333180 28366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:02.333473 28366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:02.333570 28366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:02.340754 28366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:02.340878 28366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:02.341002 28366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:58:02.350569 28366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.129.keytab
--principal=kudu/127.8.204.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:02.351877 28366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:02.353188 28366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:02.366667 28372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:02.366726 28374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:02.366786 28371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:02.367740 28366 server_base.cc:1034] running on GCE node
I20260927 22:58:02.368372 28366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:02.369354 28366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:02.370556 28366 hybrid_clock.cc:648] HybridClock initialized: now 1790549882370514 us; error 43 us; skew 500 ppm
I20260927 22:58:02.370878 28366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:02 dist-test-slave-40vx krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:02.374172 28366 init.cc:377] Logged in from keytab as kudu/127.8.204.129@KRBTEST.COM (short username kudu)
I20260927 22:58:02.375653 28366 webserver.cc:460] Webserver started at http://127.8.204.129:33041/ using document root <none> and password file <none>
I20260927 22:58:02.376343 28366 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:02.376487 28366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:02.376806 28366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:02.379798 28366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "93bc954bc1c942648b6c54e7e4704169"
format_stamp: "Formatted at 2026-09-27 22:58:02 on dist-test-slave-40vx"
server_key: "8f7cd88cce776db6aa2ae06f2e5e8e4b"
server_key_iv: "efccef064b4a9e065a89a5f75e721a6e"
server_key_version: "encryptionkey@0"
I20260927 22:58:02.380605 28366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "93bc954bc1c942648b6c54e7e4704169"
format_stamp: "Formatted at 2026-09-27 22:58:02 on dist-test-slave-40vx"
server_key: "8f7cd88cce776db6aa2ae06f2e5e8e4b"
server_key_iv: "efccef064b4a9e065a89a5f75e721a6e"
server_key_version: "encryptionkey@0"
I20260927 22:58:02.385905 28366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 22:58:02.389498 28381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:02.390908 28366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:58:02.391036 28366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "93bc954bc1c942648b6c54e7e4704169"
format_stamp: "Formatted at 2026-09-27 22:58:02 on dist-test-slave-40vx"
server_key: "8f7cd88cce776db6aa2ae06f2e5e8e4b"
server_key_iv: "efccef064b4a9e065a89a5f75e721a6e"
server_key_version: "encryptionkey@0"
I20260927 22:58:02.391201 28366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:02.405862 28366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:02.409746 28366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:02.409977 28366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:02.410852 28366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:58:02.412195 28366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:58:02.412271 28366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:02.412351 28366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:58:02.412398 28366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:02.444806 28366 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:39167
I20260927 22:58:02.444893 28493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:39167 every 8 connection(s)
I20260927 22:58:02.446141 28366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:58:02.446713 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28366
I20260927 22:58:02.446918 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:58:02.447515 9010 external_mini_cluster.cc:1428] Setting key a556f2a6e45d479c8000ca450474a461
Sep 27 22:58:02 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.129@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:02.468797 28496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:02.448943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:42863 (local address 127.8.204.190:46333)
0927 22:58:02.449572 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:02.449578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:02.450666 (+ 1088us) server_negotiation.cc:407] Connection header received
0927 22:58:02.452623 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:02.452632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:02.452724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:02.452854 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:02.454498 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.455705 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.457523 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.457956 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.458055 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:02.462262 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:02.462267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:02.462271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:02.462305 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:02.465445 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:02.466268 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:02.466274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:02.466278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:02.466416 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:02.466886 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:02.466891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:02.466894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:02.467177 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:02.467372 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:02.468156 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 22:58:02.468403 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":591,"threads_started":1}
I20260927 22:58:02.469859 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:02.449628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:46333 (local address 127.8.204.129:42863)
0927 22:58:02.450478 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:02.450529 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:02.452340 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:02.453172 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:02.453181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:02.453715 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:02.454309 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:02.454329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.455901 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.455905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:02.457365 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:02.457382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.458197 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.458207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:02.458531 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:02.459253 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:02.459313 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:02.462016 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:02.465629 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:02.465639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:02.465666 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:02.466135 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:02.466535 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:02.466540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:02.466544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:02.466759 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:02.467372 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:02.467382 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:02.467768 (+ 386us) client_negotiation.cc:769] Sending connection context
0927 22:58:02.468059 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 22:58:02.468691 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":338,"threads_started":1}
I20260927 22:58:02.471673 28494 heartbeater.cc:347] Connected to a master server at 127.8.204.190:46333
I20260927 22:58:02.471999 28494 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.204.130@KRBTEST.COM; defaulting to no policy
I20260927 22:58:02.472757 28494 heartbeater.cc:511] Master 127.8.204.190:46333 requested a full tablet report, sending...
I20260927 22:58:02.474864 28305 ts_manager.cc:194] Registered new tserver with Master: 93bc954bc1c942648b6c54e7e4704169 (127.8.204.129:39167)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.130@KRBTEST.COM" created.
I20260927 22:58:02.476629 28305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.129@KRBTEST.COM'} at 127.8.204.129:42863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
Entry for principal kudu/127.8.204.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab.
I20260927 22:58:02.505966 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:46333
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:02.829264 28500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:02.829548 28500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:02.829711 28500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:02.837919 28500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:02.838060 28500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:02.838218 28500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:58:02.853645 28500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.130.keytab
--principal=kudu/127.8.204.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:02.855159 28500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:02.856773 28500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:02.870193 28505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:02.870354 28508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:02.870162 28506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:02.870926 28500 server_base.cc:1034] running on GCE node
I20260927 22:58:02.871704 28500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:02.872663 28500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:02.873900 28500 hybrid_clock.cc:648] HybridClock initialized: now 1790549882873852 us; error 44 us; skew 500 ppm
I20260927 22:58:02.874218 28500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:02 dist-test-slave-40vx krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:02.882756 28500 init.cc:377] Logged in from keytab as kudu/127.8.204.130@KRBTEST.COM (short username kudu)
I20260927 22:58:02.884371 28500 webserver.cc:460] Webserver started at http://127.8.204.130:36001/ using document root <none> and password file <none>
I20260927 22:58:02.885051 28500 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:02.885198 28500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:02.885505 28500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:02.888368 28500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "515bcac32e75479fbee3bc4ca0856a5c"
format_stamp: "Formatted at 2026-09-27 22:58:02 on dist-test-slave-40vx"
server_key: "88515a3b3799203d59cad5e54a0b2548"
server_key_iv: "e104fe99c291744d8530aea87df37641"
server_key_version: "encryptionkey@0"
I20260927 22:58:02.889209 28500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "515bcac32e75479fbee3bc4ca0856a5c"
format_stamp: "Formatted at 2026-09-27 22:58:02 on dist-test-slave-40vx"
server_key: "88515a3b3799203d59cad5e54a0b2548"
server_key_iv: "e104fe99c291744d8530aea87df37641"
server_key_version: "encryptionkey@0"
I20260927 22:58:02.894285 28500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:58:02.897679 28515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:02.898967 28500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:58:02.899094 28500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "515bcac32e75479fbee3bc4ca0856a5c"
format_stamp: "Formatted at 2026-09-27 22:58:02 on dist-test-slave-40vx"
server_key: "88515a3b3799203d59cad5e54a0b2548"
server_key_iv: "e104fe99c291744d8530aea87df37641"
server_key_version: "encryptionkey@0"
I20260927 22:58:02.899256 28500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:02.930107 28500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:02.934273 28500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:02.934561 28500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:02.935545 28500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:58:02.936847 28500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:58:02.936918 28500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:02.937091 28500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:58:02.937142 28500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:02.970844 28500 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:38281
I20260927 22:58:02.971020 28627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:38281 every 8 connection(s)
I20260927 22:58:02.972180 28500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:58:02.979172 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28500
I20260927 22:58:02.979390 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:58:02.979907 9010 external_mini_cluster.cc:1428] Setting key a27b70111db30a1773e0ffcf60210f62
Sep 27 22:58:02 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549882, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.130@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:02.994125 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:02.974818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:41739 (local address 127.8.204.190:46333)
0927 22:58:02.975329 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:02.975336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:02.976275 (+ 939us) server_negotiation.cc:407] Connection header received
0927 22:58:02.978154 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:02.978160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:02.978253 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:02.978369 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:02.980382 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.981413 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.983819 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.984327 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.984426 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:02.988522 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:02.988527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:02.988530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:02.988559 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:02.991151 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:02.991882 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:02.991889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:02.991897 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:02.992018 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:02.992435 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:02.992440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:02.992443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:02.992693 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:02.992855 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:02.993506 (+ 651us) server_negotiation.cc:299] Negotiation successful
0927 22:58:02.993724 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260927 22:58:02.995117 28631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:02.975461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:46333 (local address 127.8.204.130:41739)
0927 22:58:02.976105 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:02.976145 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:02.977888 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:02.978595 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:02.978602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:02.979376 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:02.980174 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:02.980200 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.981580 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.981584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:02.983666 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:02.983683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:02.984549 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:02.984558 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:02.984896 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:02.985606 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:02.985638 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:02.988287 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:02.991309 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:02.991319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:02.991336 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:02.991744 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:02.992129 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:02.992134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:02.992137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:02.992319 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:02.992833 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:02.992840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:02.993164 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 22:58:02.993425 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 22:58:02.993954 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":257,"threads_started":1}
I20260927 22:58:02.996731 28628 heartbeater.cc:347] Connected to a master server at 127.8.204.190:46333
I20260927 22:58:02.997023 28628 heartbeater.cc:464] Registering TS with master...
I20260927 22:58:02.997720 28628 heartbeater.cc:511] Master 127.8.204.190:46333 requested a full tablet report, sending...
I20260927 22:58:02.999137 28305 ts_manager.cc:194] Registered new tserver with Master: 515bcac32e75479fbee3bc4ca0856a5c (127.8.204.130:38281)
I20260927 22:58:02.999989 28305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.130@KRBTEST.COM'} at 127.8.204.130:41739
WARNING: no policy specified for kudu/127.8.204.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
Entry for principal kudu/127.8.204.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab.
I20260927 22:58:03.035892 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:46333
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:41291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:03.354853 28634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:03.355118 28634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:03.355309 28634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:03.362342 28634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:03.362447 28634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:03.362637 28634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:58:03.372128 28634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:41291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.131.keytab
--principal=kudu/127.8.204.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:46333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:03.373404 28634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:03.374884 28634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:03.388204 28640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:03.388250 28642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:03.388267 28639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:03.389038 28634 server_base.cc:1034] running on GCE node
I20260927 22:58:03.389599 28634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:03.390565 28634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:03.391778 28634 hybrid_clock.cc:648] HybridClock initialized: now 1790549883391730 us; error 57 us; skew 500 ppm
I20260927 22:58:03.392076 28634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:03 dist-test-slave-40vx krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:03.395264 28634 init.cc:377] Logged in from keytab as kudu/127.8.204.131@KRBTEST.COM (short username kudu)
I20260927 22:58:03.396759 28634 webserver.cc:460] Webserver started at http://127.8.204.131:42401/ using document root <none> and password file <none>
I20260927 22:58:03.397432 28634 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:03.397558 28634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:03.397867 28634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:03.400660 28634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "7d059bd04f9749fb9e0d55a757691f7f"
format_stamp: "Formatted at 2026-09-27 22:58:03 on dist-test-slave-40vx"
server_key: "83547e361c66282d2aaf5ae81a733efb"
server_key_iv: "b61af47474c753d21fda8013ab213a47"
server_key_version: "encryptionkey@0"
I20260927 22:58:03.401402 28634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "7d059bd04f9749fb9e0d55a757691f7f"
format_stamp: "Formatted at 2026-09-27 22:58:03 on dist-test-slave-40vx"
server_key: "83547e361c66282d2aaf5ae81a733efb"
server_key_iv: "b61af47474c753d21fda8013ab213a47"
server_key_version: "encryptionkey@0"
I20260927 22:58:03.406320 28634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:58:03.409672 28649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:03.410883 28634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:58:03.411063 28634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "7d059bd04f9749fb9e0d55a757691f7f"
format_stamp: "Formatted at 2026-09-27 22:58:03 on dist-test-slave-40vx"
server_key: "83547e361c66282d2aaf5ae81a733efb"
server_key_iv: "b61af47474c753d21fda8013ab213a47"
server_key_version: "encryptionkey@0"
I20260927 22:58:03.411262 28634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:03.424888 28634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:03.428581 28634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:03.428858 28634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:03.429749 28634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:58:03.430999 28634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:58:03.431102 28634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:03.431247 28634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:58:03.431347 28634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:58:03.464002 28634 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:46479
I20260927 22:58:03.464057 28761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:46479 every 8 connection(s)
I20260927 22:58:03.465287 28634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:58:03.468142 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28634
I20260927 22:58:03.468353 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:58:03.468787 9010 external_mini_cluster.cc:1428] Setting key a97e541c364c0207008570c2305914d1
Sep 27 22:58:03 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549883, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.131@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:03.479902 28494 heartbeater.cc:503] Master 127.8.204.190:46333 was elected leader, sending a full tablet report...
I20260927 22:58:03.486133 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.468070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:37525 (local address 127.8.204.190:46333)
0927 22:58:03.468300 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.468306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.469633 (+ 1327us) server_negotiation.cc:407] Connection header received
0927 22:58:03.471960 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.471964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.472027 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.472108 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:03.473479 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.474254 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.475915 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.476299 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.476394 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.479779 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:03.479785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:03.479788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:03.479822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:03.482998 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.483798 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.483803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.483807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.483977 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.484459 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.484463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.484467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.484745 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:03.484917 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.485591 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.485821 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:58:03.487036 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.468652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:46333 (local address 127.8.204.131:37525)
0927 22:58:03.469441 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:03.469508 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:03.471664 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:03.472305 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:03.472312 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:03.472815 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:03.473329 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.473344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.474387 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.474391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.475731 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.475742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.476475 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.476479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.476700 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.477337 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:03.477376 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:03.479554 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:03.483155 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:03.483164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:03.483179 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:03.483664 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:03.484094 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:03.484099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:03.484102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:03.484334 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:03.484882 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:03.484890 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:03.485230 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 22:58:03.485499 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 22:58:03.485993 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":334,"threads_started":1}
I20260927 22:58:03.488309 28762 heartbeater.cc:347] Connected to a master server at 127.8.204.190:46333
I20260927 22:58:03.488595 28762 heartbeater.cc:464] Registering TS with master...
I20260927 22:58:03.489369 28762 heartbeater.cc:511] Master 127.8.204.190:46333 requested a full tablet report, sending...
I20260927 22:58:03.490723 28305 ts_manager.cc:194] Registered new tserver with Master: 7d059bd04f9749fb9e0d55a757691f7f (127.8.204.131:46479)
I20260927 22:58:03.491575 28305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.204.131@KRBTEST.COM'} at 127.8.204.131:37525
I20260927 22:58:03.495961 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:58:03.512750 28630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.500006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.8.204.190:46333)
0927 22:58:03.500158 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.500162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.500604 (+ 442us) server_negotiation.cc:407] Connection header received
0927 22:58:03.500798 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.500803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.500875 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.500974 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:03.502400 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.503222 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.504565 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.504971 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.505105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.507487 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:03.507492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:03.507495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:03.507524 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:03.510083 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.510742 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.510749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.510752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.510882 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.511236 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.511261 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.511264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.511592 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:03.511860 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.512211 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.512431 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:58:03.516707 28305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60982:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:58:03.523442 28305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:58:03.556947 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.534979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.8.204.130:38281)
0927 22:58:03.547452 (+ 12473us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.547462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.547496 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:58:03.547613 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.547622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.547948 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.548204 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:03.549413 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.551034 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.553924 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.555452 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.555564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.555788 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.556436 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.556611 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12262,"thread_start_us":12370,"threads_started":1}
I20260927 22:58:03.557986 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.534725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:38281 (local address 127.0.0.1:49094)
0927 22:58:03.546833 (+ 12108us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:03.546919 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:03.547120 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:03.548648 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:03.548656 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:03.548704 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:03.549188 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.549200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.551211 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.551216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.553734 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.553747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.555947 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.555953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.556157 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.556205 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:58:03.557327 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0927 22:58:03.557555 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11835,"spinlock_wait_cycles":8576,"thread_start_us":13482,"threads_started":1}
I20260927 22:58:03.558691 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.535103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39167 (local address 127.0.0.1:52150)
0927 22:58:03.548651 (+ 13548us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:03.548673 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:03.548812 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:03.549830 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:03.549836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:03.549860 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:03.550264 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.550274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.553447 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.553452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.555637 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.555651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.557904 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.557913 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.558113 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.558147 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:58:03.558235 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:58:03.558366 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13368,"thread_start_us":11295,"threads_started":1}
I20260927 22:58:03.559585 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.535267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52150 (local address 127.8.204.129:39167)
0927 22:58:03.548352 (+ 13085us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.548372 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.548966 (+ 594us) server_negotiation.cc:407] Connection header received
0927 22:58:03.549076 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.549087 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.549370 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.549570 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:03.550465 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.553297 (+ 2832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.555812 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.557320 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.557457 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.557701 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.559077 (+ 1376us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.559255 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12709,"thread_start_us":12535,"threads_started":1}
I20260927 22:58:03.562590 28563 tablet_service.cc:1467] Processing CreateTablet for tablet 80465f75aedb402f8b7130fe5335bb14 (DEFAULT_TABLE table=test-table [id=05df86af0bcf4f96af4fcf00787b198a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:58:03.562741 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 80465f75aedb402f8b7130fe5335bb14 (DEFAULT_TABLE table=test-table [id=05df86af0bcf4f96af4fcf00787b198a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:58:03.563560 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.548787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:46479 (local address 127.0.0.1:33350)
0927 22:58:03.555522 (+ 6735us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:03.555554 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:03.555712 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:03.557017 (+ 1305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:03.557021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:03.557045 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:03.557407 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.557418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.559454 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.559459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.561071 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.561085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.562765 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.562783 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.562946 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.563007 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 22:58:03.563095 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 22:58:03.563219 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6550,"thread_start_us":7019,"threads_started":1}
I20260927 22:58:03.563916 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.549116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.8.204.131:46479)
0927 22:58:03.556524 (+ 7408us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.556530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.556556 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:58:03.556631 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.556637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.556827 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.557040 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:03.557605 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.559236 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.561247 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.562542 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.562677 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.562897 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.563402 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.563561 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7237,"thread_start_us":7062,"threads_started":1}
I20260927 22:58:03.564186 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80465f75aedb402f8b7130fe5335bb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:03.564201 28563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80465f75aedb402f8b7130fe5335bb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:03.567793 28697 tablet_service.cc:1467] Processing CreateTablet for tablet 80465f75aedb402f8b7130fe5335bb14 (DEFAULT_TABLE table=test-table [id=05df86af0bcf4f96af4fcf00787b198a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:58:03.569329 28697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80465f75aedb402f8b7130fe5335bb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:03.576341 28780 tablet_bootstrap.cc:492] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Bootstrap starting.
I20260927 22:58:03.578646 28781 tablet_bootstrap.cc:492] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Bootstrap starting.
I20260927 22:58:03.580173 28780 tablet_bootstrap.cc:654] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:03.581498 28780 log.cc:824] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:03.582306 28781 tablet_bootstrap.cc:654] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:03.583453 28783 tablet_bootstrap.cc:492] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Bootstrap starting.
I20260927 22:58:03.583894 28781 log.cc:824] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:03.586647 28780 tablet_bootstrap.cc:492] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: No bootstrap required, opened a new log
I20260927 22:58:03.586826 28783 tablet_bootstrap.cc:654] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:03.586921 28781 tablet_bootstrap.cc:492] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: No bootstrap required, opened a new log
I20260927 22:58:03.586933 28780 ts_tablet_manager.cc:1397] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260927 22:58:03.587165 28781 ts_tablet_manager.cc:1397] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 22:58:03.587832 28783 log.cc:824] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:03.590363 28783 tablet_bootstrap.cc:492] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: No bootstrap required, opened a new log
I20260927 22:58:03.590620 28783 ts_tablet_manager.cc:1397] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260927 22:58:03.597836 28781 raft_consensus.cc:348] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.597836 28780 raft_consensus.cc:348] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.597913 28783 raft_consensus.cc:348] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.598258 28780 raft_consensus.cc:374] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:03.598274 28783 raft_consensus.cc:374] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:03.598379 28783 raft_consensus.cc:729] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d059bd04f9749fb9e0d55a757691f7f, State: Initialized, Role: FOLLOWER
I20260927 22:58:03.598393 28781 raft_consensus.cc:374] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:03.598424 28780 raft_consensus.cc:729] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93bc954bc1c942648b6c54e7e4704169, State: Initialized, Role: FOLLOWER
I20260927 22:58:03.598487 28781 raft_consensus.cc:729] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515bcac32e75479fbee3bc4ca0856a5c, State: Initialized, Role: FOLLOWER
I20260927 22:58:03.599149 28783 consensus_queue.cc:260] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.599149 28781 consensus_queue.cc:260] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.599149 28780 consensus_queue.cc:260] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.600333 28762 heartbeater.cc:503] Master 127.8.204.190:46333 was elected leader, sending a full tablet report...
I20260927 22:58:03.600744 28783 ts_tablet_manager.cc:1428] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:58:03.601984 28628 heartbeater.cc:503] Master 127.8.204.190:46333 was elected leader, sending a full tablet report...
I20260927 22:58:03.602208 28780 ts_tablet_manager.cc:1428] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260927 22:58:03.602562 28781 ts_tablet_manager.cc:1428] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260927 22:58:03.698815 28495 tablet.cc:2369] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:58:03.716848 28763 tablet.cc:2369] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:58:03.724247 28629 tablet.cc:2369] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:58:03.740897 28787 raft_consensus.cc:482] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:58:03.741220 28787 raft_consensus.cc:504] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.742956 28787 leader_election.cc:288] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93bc954bc1c942648b6c54e7e4704169 (127.8.204.129:39167), 7d059bd04f9749fb9e0d55a757691f7f (127.8.204.131:46479)
I20260927 22:58:03.749960 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.743626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:50007 (local address 127.8.204.131:46479)
0927 22:58:03.743863 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.743872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.744638 (+ 766us) server_negotiation.cc:407] Connection header received
0927 22:58:03.744830 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.744841 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.744901 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.744994 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:03.745612 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.746352 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.748188 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.748956 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.749132 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.749218 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.749569 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.749692 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:58:03.749995 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.743428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:47045 (local address 127.8.204.129:39167)
0927 22:58:03.743696 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.743705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.744532 (+ 827us) server_negotiation.cc:407] Connection header received
0927 22:58:03.744753 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.744757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.744822 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.744920 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:03.745551 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.746266 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.748198 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.748973 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.749161 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.749241 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.749586 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.749709 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:58:03.750138 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.743263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:46479 (local address 127.8.204.130:50007)
0927 22:58:03.744510 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:03.744527 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:03.744687 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:03.745019 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:03.745027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:03.745060 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:03.745462 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.745470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.746486 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.746492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.748034 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.748046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.749190 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.749197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.749369 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.749437 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 22:58:03.749542 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:58:03.749668 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":54,"thread_start_us":901,"threads_started":1}
I20260927 22:58:03.750181 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.743214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:39167 (local address 127.8.204.130:47045)
0927 22:58:03.744419 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:03.744434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:03.744606 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:03.744942 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:03.744946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:03.744971 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:03.745403 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.745415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.746403 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.746408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.748045 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:03.748057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.749228 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.749234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:03.749436 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.749458 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 22:58:03.749560 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:58:03.749682 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":93,"thread_start_us":774,"threads_started":1}
I20260927 22:58:03.751513 28717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80465f75aedb402f8b7130fe5335bb14" candidate_uuid: "515bcac32e75479fbee3bc4ca0856a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d059bd04f9749fb9e0d55a757691f7f" is_pre_election: true
I20260927 22:58:03.751513 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80465f75aedb402f8b7130fe5335bb14" candidate_uuid: "515bcac32e75479fbee3bc4ca0856a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93bc954bc1c942648b6c54e7e4704169" is_pre_election: true
I20260927 22:58:03.752099 28717 raft_consensus.cc:2454] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515bcac32e75479fbee3bc4ca0856a5c in term 0.
I20260927 22:58:03.752097 28449 raft_consensus.cc:2454] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515bcac32e75479fbee3bc4ca0856a5c in term 0.
I20260927 22:58:03.752830 28518 leader_election.cc:302] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515bcac32e75479fbee3bc4ca0856a5c, 93bc954bc1c942648b6c54e7e4704169; no voters:
I20260927 22:58:03.753224 28787 raft_consensus.cc:2781] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:58:03.753379 28787 raft_consensus.cc:482] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:58:03.753496 28787 raft_consensus.cc:3037] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:03.755766 28787 raft_consensus.cc:504] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.756361 28787 leader_election.cc:288] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [CANDIDATE]: Term 1 election: Requested vote from peers 93bc954bc1c942648b6c54e7e4704169 (127.8.204.129:39167), 7d059bd04f9749fb9e0d55a757691f7f (127.8.204.131:46479)
I20260927 22:58:03.756829 28717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80465f75aedb402f8b7130fe5335bb14" candidate_uuid: "515bcac32e75479fbee3bc4ca0856a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d059bd04f9749fb9e0d55a757691f7f"
I20260927 22:58:03.756879 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80465f75aedb402f8b7130fe5335bb14" candidate_uuid: "515bcac32e75479fbee3bc4ca0856a5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93bc954bc1c942648b6c54e7e4704169"
I20260927 22:58:03.757018 28717 raft_consensus.cc:3037] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:03.757126 28449 raft_consensus.cc:3037] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:03.759207 28717 raft_consensus.cc:2454] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515bcac32e75479fbee3bc4ca0856a5c in term 1.
I20260927 22:58:03.759423 28449 raft_consensus.cc:2454] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515bcac32e75479fbee3bc4ca0856a5c in term 1.
I20260927 22:58:03.759653 28517 leader_election.cc:302] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515bcac32e75479fbee3bc4ca0856a5c, 7d059bd04f9749fb9e0d55a757691f7f; no voters:
I20260927 22:58:03.759938 28787 raft_consensus.cc:2781] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:03.760350 28787 raft_consensus.cc:686] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 1 LEADER]: Becoming Leader. State: Replica: 515bcac32e75479fbee3bc4ca0856a5c, State: Running, Role: LEADER
I20260927 22:58:03.760880 28787 consensus_queue.cc:237] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } }
I20260927 22:58:03.766099 28304 catalog_manager.cc:5495] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 515bcac32e75479fbee3bc4ca0856a5c (127.8.204.130). New cstate: current_term: 1 leader_uuid: "515bcac32e75479fbee3bc4ca0856a5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515bcac32e75479fbee3bc4ca0856a5c" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 38281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 } health_report { overall_health: UNKNOWN } } }
I20260927 22:58:03.783456 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.778170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.8.204.130:38281)
0927 22:58:03.778404 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.778410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.778454 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:58:03.778604 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.778608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.778679 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.778842 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:03.779181 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.779909 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.781226 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.781619 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.781751 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.782213 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:03.782762 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:03.782955 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.783042 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.783118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:58:03.791865 28449 raft_consensus.cc:1261] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 1 FOLLOWER]: Refusing update from remote peer 515bcac32e75479fbee3bc4ca0856a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:03.791865 28717 raft_consensus.cc:1261] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 1 FOLLOWER]: Refusing update from remote peer 515bcac32e75479fbee3bc4ca0856a5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:03.793000 28791 consensus_queue.cc:1035] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93bc954bc1c942648b6c54e7e4704169" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 39167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:03.793599 28787 consensus_queue.cc:1035] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d059bd04f9749fb9e0d55a757691f7f" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 46479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:03.804389 28794 mvcc.cc:205] Tried to move back new op lower bound from 7334092323999887360 to 7334092323888295936. Current Snapshot: MvccSnapshot[applied={T|T < 7334092323999887360}]
I20260927 22:58:03.815529 28796 mvcc.cc:205] Tried to move back new op lower bound from 7334092323999887360 to 7334092323888295936. Current Snapshot: MvccSnapshot[applied={T|T < 7334092323999887360 or (T in {7334092323999887360})}]
I20260927 22:58:03.820722 28305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:58:03.821018 28305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60982:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:58:03.824615 28305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d51e7f249e16449785c060cb63b64ab8: Sending DeleteTablet for 3 replicas of tablet 80465f75aedb402f8b7130fe5335bb14
I20260927 22:58:03.825742 28429 tablet_service.cc:1514] Processing DeleteTablet for tablet 80465f75aedb402f8b7130fe5335bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:58:03 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:52150
I20260927 22:58:03.825814 28563 tablet_service.cc:1514] Processing DeleteTablet for tablet 80465f75aedb402f8b7130fe5335bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:58:03 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:49094
I20260927 22:58:03.825912 28697 tablet_service.cc:1514] Processing DeleteTablet for tablet 80465f75aedb402f8b7130fe5335bb14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:58:03 UTC) from {username='kudu', principal='kudu/127.8.204.190@KRBTEST.COM'} at 127.0.0.1:33350
I20260927 22:58:03.827817 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28366
I20260927 22:58:03.839672 28804 tablet_replica.cc:330] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: stopping tablet replica
I20260927 22:58:03.839749 28803 tablet_replica.cc:330] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: stopping tablet replica
I20260927 22:58:03.840238 28803 raft_consensus.cc:2229] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:58:03.840339 28805 tablet_replica.cc:330] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: stopping tablet replica
I20260927 22:58:03.840955 28805 raft_consensus.cc:2229] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:58:03.841459 28804 raft_consensus.cc:2229] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:58:03.841598 28803 raft_consensus.cc:2258] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:58:03.842060 28805 raft_consensus.cc:2258] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:58:03.842733 28804 raft_consensus.cc:2258] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:58:03.844277 28803 ts_tablet_manager.cc:1905] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:58:03.845127 28805 ts_tablet_manager.cc:1905] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:58:03.846694 28804 ts_tablet_manager.cc:1905] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:58:03 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.129@KRBTEST.COM
I20260927 22:58:03.851155 28803 ts_tablet_manager.cc:1918] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:58:03.851428 28803 log.cc:1196] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/80465f75aedb402f8b7130fe5335bb14
I20260927 22:58:03.852051 28803 ts_tablet_manager.cc:1939] T 80465f75aedb402f8b7130fe5335bb14 P 93bc954bc1c942648b6c54e7e4704169: Deleting consensus metadata
I20260927 22:58:03.857380 28291 catalog_manager.cc:4841] TS 93bc954bc1c942648b6c54e7e4704169 (127.8.204.129:39167): tablet 80465f75aedb402f8b7130fe5335bb14 (table test-table [id=05df86af0bcf4f96af4fcf00787b198a]) successfully deleted
I20260927 22:58:03.858521 28805 ts_tablet_manager.cc:1918] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:58:03.858695 28805 log.cc:1196] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/80465f75aedb402f8b7130fe5335bb14
I20260927 22:58:03.859146 28805 ts_tablet_manager.cc:1939] T 80465f75aedb402f8b7130fe5335bb14 P 7d059bd04f9749fb9e0d55a757691f7f: Deleting consensus metadata
I20260927 22:58:03.860898 28290 catalog_manager.cc:4841] TS 7d059bd04f9749fb9e0d55a757691f7f (127.8.204.131:46479): tablet 80465f75aedb402f8b7130fe5335bb14 (table test-table [id=05df86af0bcf4f96af4fcf00787b198a]) successfully deleted
I20260927 22:58:03.861079 28804 ts_tablet_manager.cc:1918] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:58:03.861289 28804 log.cc:1196] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/80465f75aedb402f8b7130fe5335bb14
I20260927 22:58:03.861874 28804 ts_tablet_manager.cc:1939] T 80465f75aedb402f8b7130fe5335bb14 P 515bcac32e75479fbee3bc4ca0856a5c: Deleting consensus metadata
I20260927 22:58:03.863605 28290 catalog_manager.cc:4841] TS 515bcac32e75479fbee3bc4ca0856a5c (127.8.204.130:38281): tablet 80465f75aedb402f8b7130fe5335bb14 (table test-table [id=05df86af0bcf4f96af4fcf00787b198a]) successfully deleted
I20260927 22:58:03.864674 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.829084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.8.204.129:39167)
0927 22:58:03.829311 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.829321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.840865 (+ 11544us) server_negotiation.cc:407] Connection header received
0927 22:58:03.841064 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.841069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.841161 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.841270 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:03.842985 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.844121 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.845947 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.846609 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.846709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.851925 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:03.851940 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:03.851957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:03.851993 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:03.855754 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.862226 (+ 6472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.862235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.862240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.862414 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.862929 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.862936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.862939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.863420 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:03.863721 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.864028 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.864289 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:58:03.865374 28489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:03.955461 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28366
I20260927 22:58:03.975363 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28500
Sep 27 22:58:03 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.130@KRBTEST.COM
I20260927 22:58:03.996325 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:03.976644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49110 (local address 127.8.204.130:38281)
0927 22:58:03.976843 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:03.976849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:03.976867 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:58:03.976924 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:03.976929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:03.977010 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:03.977096 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:03.978778 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.980026 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.982101 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:03.982797 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:03.982929 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:03.988868 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:03.988884 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:03.988907 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:03.988961 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:03.993258 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.994058 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.994067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.994071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.994237 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:03.994640 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:03.994645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:03.994648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:03.995088 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:03.995405 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:03.995722 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 22:58:03.995944 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:58:03.997208 28623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:04.110924 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28500
I20260927 22:58:04.130278 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28634
Sep 27 22:58:04 dist-test-slave-40vx krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.131@KRBTEST.COM
I20260927 22:58:04.144850 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:04.131029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.8.204.131:46479)
0927 22:58:04.131200 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:04.131204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:04.131222 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:58:04.131406 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:04.131410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:04.131472 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:04.131556 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:04.132914 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:04.133637 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:04.134975 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:04.135463 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:04.135562 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:04.139940 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:04.139955 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:04.139971 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:04.140010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:04.142513 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:04.143097 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:04.143104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:04.143106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:04.143264 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:04.143646 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:04.143650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:04.143652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:04.143989 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:04.144195 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:04.144442 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:58:04.144598 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:58:04.145450 28757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:04.233912 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28634
I20260927 22:58:04.250311 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28273
I20260927 22:58:04.251392 28335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:04.362207 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28273
2026-09-27T22:58:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4324 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:04Z Disabled control of system clock
I20260927 22:58:04.440341 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37289
--webserver_interface=127.8.204.190
--webserver_port=0
--builtin_ntp_servers=127.8.204.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:58:04.770709 28814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:04.770977 28814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:04.771131 28814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:04.778507 28814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:04.778618 28814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:04.778710 28814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:04.778782 28814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:04.778846 28814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:04.788316 28814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:04.789613 28814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:04.791457 28814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:04.803361 28822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:04.803436 28820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:04.803417 28819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:04.804209 28814 server_base.cc:1034] running on GCE node
I20260927 22:58:04.804966 28814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:04.806411 28814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:04.807637 28814 hybrid_clock.cc:648] HybridClock initialized: now 1790549884807597 us; error 36 us; skew 500 ppm
I20260927 22:58:04.808024 28814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:58:04.810345 28814 webserver.cc:460] Webserver started at http://127.8.204.190:45463/ using document root <none> and password file <none>
I20260927 22:58:04.811051 28814 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:04.811187 28814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:04.811534 28814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:04.814435 28814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "c5b2ef5334814e98a88eb12c429d3988"
format_stamp: "Formatted at 2026-09-27 22:58:04 on dist-test-slave-40vx"
server_key: "e7cd5b83d0ff84032269b36d15db1a2e"
server_key_iv: "d6e84dca715e6988a70f2888a5e74c87"
server_key_version: "encryptionkey@0"
I20260927 22:58:04.815240 28814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "c5b2ef5334814e98a88eb12c429d3988"
format_stamp: "Formatted at 2026-09-27 22:58:04 on dist-test-slave-40vx"
server_key: "e7cd5b83d0ff84032269b36d15db1a2e"
server_key_iv: "d6e84dca715e6988a70f2888a5e74c87"
server_key_version: "encryptionkey@0"
I20260927 22:58:04.820528 28814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 22:58:04.823885 28828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:04.825117 28814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:04.825304 28814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "c5b2ef5334814e98a88eb12c429d3988"
format_stamp: "Formatted at 2026-09-27 22:58:04 on dist-test-slave-40vx"
server_key: "e7cd5b83d0ff84032269b36d15db1a2e"
server_key_iv: "d6e84dca715e6988a70f2888a5e74c87"
server_key_version: "encryptionkey@0"
I20260927 22:58:04.825516 28814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:04.849411 28814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:04.850311 28814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:04.850608 28814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:04.869285 28814 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:37289
I20260927 22:58:04.869298 28879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:37289 every 8 connection(s)
I20260927 22:58:04.870630 28814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:04.873406 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28814
I20260927 22:58:04.873584 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:04.874013 9010 external_mini_cluster.cc:1428] Setting key cde771a9fad5ae29084399473ff13004
I20260927 22:58:04.874434 28880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:04.889148 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:04.875904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.8.204.190:37289)
0927 22:58:04.876935 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:04.876950 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:04.876997 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:58:04.879192 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:04.879220 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:04.879839 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:04.880239 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:04.880817 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:04.882775 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:04.884113 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:04.885147 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:04.885342 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:04.886408 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:04.886417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:04.886434 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:04.887170 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:04.887304 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:04.887331 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:04.887429 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:04.887636 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 22:58:04.887943 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":347,"threads_started":1}
I20260927 22:58:04.889400 28880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988: Bootstrap starting.
I20260927 22:58:04.892575 28880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:04.893482 28880 log.cc:824] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:04.896212 28880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988: No bootstrap required, opened a new log
I20260927 22:58:04.903458 28880 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ef5334814e98a88eb12c429d3988" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37289 } }
I20260927 22:58:04.903790 28880 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:04.903860 28880 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b2ef5334814e98a88eb12c429d3988, State: Initialized, Role: FOLLOWER
I20260927 22:58:04.904386 28880 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ef5334814e98a88eb12c429d3988" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37289 } }
I20260927 22:58:04.904687 28880 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:04.904863 28880 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:04.905059 28880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:04.907480 28880 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ef5334814e98a88eb12c429d3988" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37289 } }
I20260927 22:58:04.908047 28880 leader_election.cc:302] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5b2ef5334814e98a88eb12c429d3988; no voters:
I20260927 22:58:04.908603 28880 leader_election.cc:288] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:04.908747 28885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:04.909458 28885 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [term 1 LEADER]: Becoming Leader. State: Replica: c5b2ef5334814e98a88eb12c429d3988, State: Running, Role: LEADER
I20260927 22:58:04.910053 28885 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ef5334814e98a88eb12c429d3988" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37289 } }
I20260927 22:58:04.910588 28880 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:04.912519 28887 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5b2ef5334814e98a88eb12c429d3988. Latest consensus state: current_term: 1 leader_uuid: "c5b2ef5334814e98a88eb12c429d3988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ef5334814e98a88eb12c429d3988" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37289 } } }
I20260927 22:58:04.912923 28887 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:04.913146 28886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5b2ef5334814e98a88eb12c429d3988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b2ef5334814e98a88eb12c429d3988" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37289 } } }
I20260927 22:58:04.913630 28886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:04.922955 28890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:04.929096 28890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:04.938802 28890 catalog_manager.cc:1347] Generated new cluster ID: 24e96e3460a04a9e8c715013d7491b5b
I20260927 22:58:04.938938 28890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:04.956619 28890 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:04.958470 28890 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:04.966817 28890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988: Generated new TSK 0
I20260927 22:58:04.967703 28890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:04.974001 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.129:0
--local_ip_for_outbound_sockets=127.8.204.129
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37289
--builtin_ntp_servers=127.8.204.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:58:05.322564 28905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:05.322868 28905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:05.323014 28905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:05.331005 28905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:05.331166 28905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:05.331408 28905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.129
I20260927 22:58:05.341044 28905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.204.129
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.129
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:05.342344 28905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:05.344161 28905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:05.356889 28910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:05.356840 28911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:05.356896 28913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:05.357517 28905 server_base.cc:1034] running on GCE node
I20260927 22:58:05.358095 28905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:05.358989 28905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:05.360230 28905 hybrid_clock.cc:648] HybridClock initialized: now 1790549885360154 us; error 68 us; skew 500 ppm
I20260927 22:58:05.360549 28905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:58:05.362700 28905 webserver.cc:460] Webserver started at http://127.8.204.129:45879/ using document root <none> and password file <none>
I20260927 22:58:05.363466 28905 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:05.363592 28905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:05.363931 28905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:05.366736 28905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data/instance:
uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb"
format_stamp: "Formatted at 2026-09-27 22:58:05 on dist-test-slave-40vx"
server_key: "1453d6c657536ccf1bf46944846c93ee"
server_key_iv: "628022c48abadcd9cc057278c58d614d"
server_key_version: "encryptionkey@0"
I20260927 22:58:05.367576 28905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance:
uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb"
format_stamp: "Formatted at 2026-09-27 22:58:05 on dist-test-slave-40vx"
server_key: "1453d6c657536ccf1bf46944846c93ee"
server_key_iv: "628022c48abadcd9cc057278c58d614d"
server_key_version: "encryptionkey@0"
I20260927 22:58:05.372346 28905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:58:05.375950 28919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:05.377185 28905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:05.377353 28905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal
uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb"
format_stamp: "Formatted at 2026-09-27 22:58:05 on dist-test-slave-40vx"
server_key: "1453d6c657536ccf1bf46944846c93ee"
server_key_iv: "628022c48abadcd9cc057278c58d614d"
server_key_version: "encryptionkey@0"
I20260927 22:58:05.377563 28905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:05.405851 28905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:05.406711 28905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:05.407019 28905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:05.408143 28905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:58:05.409592 28905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:58:05.409719 28905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:05.409868 28905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:58:05.409961 28905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:05.443842 28905 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.129:36637
I20260927 22:58:05.443933 29031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.129:36637 every 8 connection(s)
I20260927 22:58:05.445134 28905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/data/info.pb
I20260927 22:58:05.455677 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 28905
I20260927 22:58:05.455865 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal/instance
I20260927 22:58:05.456292 9010 external_mini_cluster.cc:1428] Setting key 3e79fcec7d7946e531de436eae46b9c4
I20260927 22:58:05.457355 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:05.447675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:34989 (local address 127.8.204.190:37289)
0927 22:58:05.448157 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:05.448162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:05.449153 (+ 991us) server_negotiation.cc:407] Connection header received
0927 22:58:05.450967 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:05.450972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:05.451063 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:05.451180 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:05.452257 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.452998 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.454498 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.454879 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.455012 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:05.456485 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:05.456491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:05.456496 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:05.456523 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:05.456559 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:05.456571 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:05.456675 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:05.456893 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 22:58:05.456989 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":273,"threads_started":1}
I20260927 22:58:05.458277 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:05.448169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37289 (local address 127.8.204.129:34989)
0927 22:58:05.449006 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:05.449042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:05.450693 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:05.451438 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:05.451446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:05.451532 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:05.452088 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:05.452103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.453142 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.453149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:05.454349 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:05.454360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.455096 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.455104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:05.455427 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:05.456177 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:58:05.456179 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:05.456217 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:58:05.456291 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:05.456674 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:05.456686 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:05.456696 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:58:05.456883 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 22:58:05.457243 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260927 22:58:05.459124 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.130:0
--local_ip_for_outbound_sockets=127.8.204.130
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37289
--builtin_ntp_servers=127.8.204.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:58:05.459877 29032 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37289
I20260927 22:58:05.460152 29032 heartbeater.cc:464] Registering TS with master...
I20260927 22:58:05.460839 29032 heartbeater.cc:511] Master 127.8.204.190:37289 requested a full tablet report, sending...
I20260927 22:58:05.462801 28845 ts_manager.cc:194] Registered new tserver with Master: 135fe6a6543b4fa2a0ca1545ea3ab9fb (127.8.204.129:36637)
I20260927 22:58:05.464350 28845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.129:34989
W20260927 22:58:05.782600 29036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:05.782866 29036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:05.782996 29036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:05.790501 29036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:05.790599 29036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:05.790762 29036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.130
I20260927 22:58:05.799861 29036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.204.130
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.130
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:05.801079 29036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:05.802866 29036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:05.818882 29041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:05.818979 29042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:05.818928 29044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:05.820137 29036 server_base.cc:1034] running on GCE node
I20260927 22:58:05.820693 29036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:05.821609 29036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:05.822835 29036 hybrid_clock.cc:648] HybridClock initialized: now 1790549885822787 us; error 70 us; skew 500 ppm
I20260927 22:58:05.823148 29036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:58:05.825335 29036 webserver.cc:460] Webserver started at http://127.8.204.130:39663/ using document root <none> and password file <none>
I20260927 22:58:05.826004 29036 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:05.826130 29036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:05.826445 29036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:05.829371 29036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data/instance:
uuid: "e1be4d2154e149db9da40e4368806c5e"
format_stamp: "Formatted at 2026-09-27 22:58:05 on dist-test-slave-40vx"
server_key: "3252aa3662e22e8b9135fdacb032c0b2"
server_key_iv: "4b8d7b4df8e2674d21456b23bb5ad4c2"
server_key_version: "encryptionkey@0"
I20260927 22:58:05.830194 29036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance:
uuid: "e1be4d2154e149db9da40e4368806c5e"
format_stamp: "Formatted at 2026-09-27 22:58:05 on dist-test-slave-40vx"
server_key: "3252aa3662e22e8b9135fdacb032c0b2"
server_key_iv: "4b8d7b4df8e2674d21456b23bb5ad4c2"
server_key_version: "encryptionkey@0"
I20260927 22:58:05.835224 29036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 22:58:05.838702 29050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:05.840094 29036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:05.840266 29036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal
uuid: "e1be4d2154e149db9da40e4368806c5e"
format_stamp: "Formatted at 2026-09-27 22:58:05 on dist-test-slave-40vx"
server_key: "3252aa3662e22e8b9135fdacb032c0b2"
server_key_iv: "4b8d7b4df8e2674d21456b23bb5ad4c2"
server_key_version: "encryptionkey@0"
I20260927 22:58:05.840481 29036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:05.860769 29036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:05.861543 29036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:05.861850 29036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:05.862766 29036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:58:05.864203 29036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:58:05.864322 29036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:05.864456 29036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:58:05.864549 29036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:05.902058 29036 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.130:39991
I20260927 22:58:05.902145 29162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.130:39991 every 8 connection(s)
I20260927 22:58:05.903326 29036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/data/info.pb
I20260927 22:58:05.903641 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29036
I20260927 22:58:05.903815 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal/instance
I20260927 22:58:05.904368 9010 external_mini_cluster.cc:1428] Setting key 1878801c48c804a1bb1fd7869a18ea98
I20260927 22:58:05.907392 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.204.131:0
--local_ip_for_outbound_sockets=127.8.204.131
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37289
--builtin_ntp_servers=127.8.204.148:34507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:58:05.916011 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:05.906109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.130:47551 (local address 127.8.204.190:37289)
0927 22:58:05.906317 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:05.906322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:05.907847 (+ 1525us) server_negotiation.cc:407] Connection header received
0927 22:58:05.909581 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:05.909588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:05.909684 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:05.909794 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:05.910958 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.911840 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.913360 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.913742 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.913884 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:05.915196 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:05.915201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:05.915204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:05.915222 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:05.915251 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:05.915263 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:05.915386 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:05.915599 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 22:58:05.915693 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:58:05.916971 29165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:05.906603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37289 (local address 127.8.204.130:47551)
0927 22:58:05.907656 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:05.907717 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:05.909323 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:05.910024 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:05.910032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:05.910120 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:05.910797 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:05.910817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.911985 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.911990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:05.913197 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:05.913209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:05.913967 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:05.913974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:05.914194 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:05.914883 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:58:05.914885 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:05.914922 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:58:05.914998 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:05.915386 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:05.915399 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:05.915409 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:58:05.915588 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 22:58:05.915964 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":331,"threads_started":1}
I20260927 22:58:05.918368 29163 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37289
I20260927 22:58:05.918676 29163 heartbeater.cc:464] Registering TS with master...
I20260927 22:58:05.919486 29163 heartbeater.cc:511] Master 127.8.204.190:37289 requested a full tablet report, sending...
I20260927 22:58:05.920979 28845 ts_manager.cc:194] Registered new tserver with Master: e1be4d2154e149db9da40e4368806c5e (127.8.204.130:39991)
I20260927 22:58:05.922250 28845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.130:47551
W20260927 22:58:06.228715 29166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:06.228971 29166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:06.229108 29166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:06.236058 29166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:06.236141 29166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:06.236235 29166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.204.131
I20260927 22:58:06.244973 29166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:34507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.204.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.204.131
--webserver_port=0
--tserver_master_addrs=127.8.204.190:37289
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.204.131
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:06.246065 29166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:06.247711 29166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:06.260030 29172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:06.260078 29174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:06.260111 29171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:06.261194 29166 server_base.cc:1034] running on GCE node
I20260927 22:58:06.261723 29166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:06.262578 29166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:06.263768 29166 hybrid_clock.cc:648] HybridClock initialized: now 1790549886263722 us; error 45 us; skew 500 ppm
I20260927 22:58:06.264088 29166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:58:06.267256 29166 webserver.cc:460] Webserver started at http://127.8.204.131:44857/ using document root <none> and password file <none>
I20260927 22:58:06.268009 29166 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:06.268124 29166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:06.268453 29166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:06.271338 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data/instance:
uuid: "69259272343f4221b590cae4eb5a5c00"
format_stamp: "Formatted at 2026-09-27 22:58:06 on dist-test-slave-40vx"
server_key: "43c0a4ed1b6cc9e6c3bb7a8fabb62059"
server_key_iv: "175cb304cbae003151ef9ff1e5670edf"
server_key_version: "encryptionkey@0"
I20260927 22:58:06.272120 29166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance:
uuid: "69259272343f4221b590cae4eb5a5c00"
format_stamp: "Formatted at 2026-09-27 22:58:06 on dist-test-slave-40vx"
server_key: "43c0a4ed1b6cc9e6c3bb7a8fabb62059"
server_key_iv: "175cb304cbae003151ef9ff1e5670edf"
server_key_version: "encryptionkey@0"
I20260927 22:58:06.277053 29166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:58:06.280498 29180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:06.281912 29166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:06.282080 29166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal
uuid: "69259272343f4221b590cae4eb5a5c00"
format_stamp: "Formatted at 2026-09-27 22:58:06 on dist-test-slave-40vx"
server_key: "43c0a4ed1b6cc9e6c3bb7a8fabb62059"
server_key_iv: "175cb304cbae003151ef9ff1e5670edf"
server_key_version: "encryptionkey@0"
I20260927 22:58:06.282343 29166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:06.305318 29166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:06.306180 29166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:06.306499 29166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:06.307509 29166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:58:06.309010 29166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:58:06.309132 29166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:06.309262 29166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:58:06.309348 29166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:06.342502 29166 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.131:36283
I20260927 22:58:06.342590 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.131:36283 every 8 connection(s)
I20260927 22:58:06.343986 29166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/data/info.pb
I20260927 22:58:06.352624 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29166
I20260927 22:58:06.352847 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal/instance
I20260927 22:58:06.353375 9010 external_mini_cluster.cc:1428] Setting key 69ea8ec73146e3cce99150a5819c0a73
I20260927 22:58:06.357174 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.346594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.131:60719 (local address 127.8.204.190:37289)
0927 22:58:06.346810 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.346816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.348299 (+ 1483us) server_negotiation.cc:407] Connection header received
0927 22:58:06.350054 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.350058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.350127 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.350212 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:06.351205 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.352005 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.354204 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.354766 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.354871 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.356474 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:06.356480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:06.356485 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:06.356512 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:06.356542 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:06.356555 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:06.356624 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.356867 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.356943 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:58:06.358094 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.347173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37289 (local address 127.8.204.131:60719)
0927 22:58:06.348124 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:06.348165 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:06.349816 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:06.350419 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:06.350426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:06.350514 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:06.351046 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.351062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.352144 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.352148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.354053 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.354068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.354989 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.354998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.355322 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.356078 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:58:06.356080 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:06.356137 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:58:06.356271 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:06.356642 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:06.356650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:06.356655 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:58:06.356818 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 22:58:06.357091 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":277,"threads_started":1}
I20260927 22:58:06.359427 29293 heartbeater.cc:347] Connected to a master server at 127.8.204.190:37289
I20260927 22:58:06.359766 29293 heartbeater.cc:464] Registering TS with master...
I20260927 22:58:06.360451 29293 heartbeater.cc:511] Master 127.8.204.190:37289 requested a full tablet report, sending...
I20260927 22:58:06.361742 28845 ts_manager.cc:194] Registered new tserver with Master: 69259272343f4221b590cae4eb5a5c00 (127.8.204.131:36283)
I20260927 22:58:06.362533 28845 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.204.131:60719
I20260927 22:58:06.368592 9010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:58:06.378628 29034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.372735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.8.204.190:37289)
0927 22:58:06.372959 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.372968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.373288 (+ 320us) server_negotiation.cc:407] Connection header received
0927 22:58:06.373492 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.373498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.373583 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.373691 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:06.374203 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.374924 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.376342 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.376745 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.376884 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.377916 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:06.377926 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:06.377929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:06.377948 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:06.377980 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:06.377992 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:06.378088 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.378234 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.378337 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:58:06.382364 28845 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:58:06.389446 28845 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:58:06.414428 29309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.403367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:39991 (local address 127.0.0.1:53060)
0927 22:58:06.405674 (+ 2307us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:06.405703 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:06.405853 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:06.406831 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:06.406838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:06.406881 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:06.407416 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.407433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.409146 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.409154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.411414 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.411428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.413435 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.413441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.413626 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.413678 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 22:58:06.413796 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:58:06.414021 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2164,"mutex_wait_us":136,"spinlock_wait_cycles":2944,"thread_start_us":2021,"threads_started":1}
I20260927 22:58:06.414739 29305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.400991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.129:36637 (local address 127.0.0.1:45982)
0927 22:58:06.402330 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:06.402396 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:06.402599 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:06.407189 (+ 4590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:06.407193 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:06.407220 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:06.407795 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.407808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.409634 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.409639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.411998 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.412013 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.413860 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.413867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.414070 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.414103 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:58:06.414210 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 22:58:06.414338 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1016,"mutex_wait_us":103,"thread_start_us":886,"threads_started":1}
I20260927 22:58:06.415071 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.401180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.8.204.129:36637)
0927 22:58:06.405410 (+ 4230us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.405423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.405461 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:58:06.405581 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.405592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.405900 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.407186 (+ 1286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:06.407992 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.409492 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.412185 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.413623 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.413794 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.414038 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.414266 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.414441 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4010,"thread_start_us":3892,"threads_started":1}
I20260927 22:58:06.415293 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.404083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.8.204.130:39991)
0927 22:58:06.406349 (+ 2266us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.406361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.406385 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:58:06.406455 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.406460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.406624 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.406831 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:06.407662 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.409012 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.411592 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.413248 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.413390 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.413686 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.414702 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.414899 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2118,"thread_start_us":1993,"threads_started":1}
I20260927 22:58:06.417124 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.401718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.8.204.131:36283)
0927 22:58:06.406397 (+ 4679us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.406403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.406427 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:58:06.406494 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.406499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.406658 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.406844 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:06.407837 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.409492 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.413138 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.414529 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.414941 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.415204 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.416575 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.416759 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4525,"thread_start_us":4366,"threads_started":1}
I20260927 22:58:06.417972 29307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.401614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36283 (local address 127.0.0.1:51602)
0927 22:58:06.403318 (+ 1704us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:06.403345 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:06.403493 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:06.406836 (+ 3343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:06.406840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:06.406890 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:06.407531 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.407544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.410121 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.410126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.412954 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.412966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.415364 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.415372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.416365 (+ 993us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.416399 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 22:58:06.416490 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:58:06.417602 (+ 1112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1509,"mutex_wait_us":170,"spinlock_wait_cycles":3840,"thread_start_us":1320,"threads_started":1}
I20260927 22:58:06.418816 28967 tablet_service.cc:1467] Processing CreateTablet for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 (DEFAULT_TABLE table=test-table [id=c5c1f12a92064a078c585f0521c06590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:58:06.418821 29098 tablet_service.cc:1467] Processing CreateTablet for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 (DEFAULT_TABLE table=test-table [id=c5c1f12a92064a078c585f0521c06590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:58:06.420006 28967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:06.420074 29098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:06.422603 29228 tablet_service.cc:1467] Processing CreateTablet for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 (DEFAULT_TABLE table=test-table [id=c5c1f12a92064a078c585f0521c06590]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:58:06.424304 29228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:06.434192 29312 tablet_bootstrap.cc:492] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Bootstrap starting.
I20260927 22:58:06.437938 29311 tablet_bootstrap.cc:492] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Bootstrap starting.
I20260927 22:58:06.438097 29313 tablet_bootstrap.cc:492] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Bootstrap starting.
I20260927 22:58:06.438620 29312 tablet_bootstrap.cc:654] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:06.440836 29312 log.cc:824] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:06.441469 29313 tablet_bootstrap.cc:654] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:06.442279 29311 tablet_bootstrap.cc:654] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:06.442555 29313 log.cc:824] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:06.443539 29311 log.cc:824] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:06.445770 29312 tablet_bootstrap.cc:492] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: No bootstrap required, opened a new log
I20260927 22:58:06.446082 29313 tablet_bootstrap.cc:492] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: No bootstrap required, opened a new log
I20260927 22:58:06.446107 29312 ts_tablet_manager.cc:1397] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:58:06.446375 29313 ts_tablet_manager.cc:1397] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260927 22:58:06.452756 29311 tablet_bootstrap.cc:492] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: No bootstrap required, opened a new log
I20260927 22:58:06.453102 29311 ts_tablet_manager.cc:1397] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260927 22:58:06.457778 29312 raft_consensus.cc:348] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.458411 29312 raft_consensus.cc:374] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:06.458562 29312 raft_consensus.cc:729] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1be4d2154e149db9da40e4368806c5e, State: Initialized, Role: FOLLOWER
I20260927 22:58:06.458390 29313 raft_consensus.cc:348] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.458879 29313 raft_consensus.cc:374] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:06.459024 29313 raft_consensus.cc:729] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69259272343f4221b590cae4eb5a5c00, State: Initialized, Role: FOLLOWER
I20260927 22:58:06.459457 29312 consensus_queue.cc:260] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.459672 29313 consensus_queue.cc:260] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.460780 29163 heartbeater.cc:503] Master 127.8.204.190:37289 was elected leader, sending a full tablet report...
I20260927 22:58:06.461197 29293 heartbeater.cc:503] Master 127.8.204.190:37289 was elected leader, sending a full tablet report...
I20260927 22:58:06.461899 29313 ts_tablet_manager.cc:1428] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260927 22:58:06.462019 29312 ts_tablet_manager.cc:1428] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260927 22:58:06.464260 29311 raft_consensus.cc:348] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.464659 29311 raft_consensus.cc:374] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:06.464767 29311 raft_consensus.cc:729] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135fe6a6543b4fa2a0ca1545ea3ab9fb, State: Initialized, Role: FOLLOWER
I20260927 22:58:06.465372 29311 consensus_queue.cc:260] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.466651 29032 heartbeater.cc:503] Master 127.8.204.190:37289 was elected leader, sending a full tablet report...
I20260927 22:58:06.467142 29311 ts_tablet_manager.cc:1428] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Time spent starting tablet: real 0.014s user 0.008s sys 0.003s
W20260927 22:58:06.595875 29294 tablet.cc:2369] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:58:06.655491 29164 tablet.cc:2369] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:58:06.692425 29319 raft_consensus.cc:482] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:58:06.692723 29319 raft_consensus.cc:504] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.694423 29319 leader_election.cc:288] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69259272343f4221b590cae4eb5a5c00 (127.8.204.131:36283), e1be4d2154e149db9da40e4368806c5e (127.8.204.130:39991)
W20260927 22:58:06.697541 29033 tablet.cc:2369] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:58:06.701856 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.695019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:33071 (local address 127.8.204.131:36283)
0927 22:58:06.695232 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.695236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.695805 (+ 569us) server_negotiation.cc:407] Connection header received
0927 22:58:06.696138 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.696144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.696233 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.696302 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:06.697060 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.698096 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.700152 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.700914 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.701111 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.701180 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.701481 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.701608 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:58:06.701892 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.696315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.204.129:58923 (local address 127.8.204.130:39991)
0927 22:58:06.696477 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.696482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.696589 (+ 107us) server_negotiation.cc:407] Connection header received
0927 22:58:06.696765 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.696768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.696829 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.696896 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:58:06.697458 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.698262 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.700173 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.700940 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.701123 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.701195 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.701513 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.701641 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:58:06.701998 29320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.695025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.131:36283 (local address 127.8.204.129:33071)
0927 22:58:06.695706 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:06.695727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:06.695950 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:06.696499 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:06.696505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:06.696539 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:06.696941 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.696952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.698252 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.698258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.699956 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.699968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.701121 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.701127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.701299 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.701344 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:58:06.701443 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:58:06.701584 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":76,"spinlock_wait_cycles":4864,"thread_start_us":377,"threads_started":1}
I20260927 22:58:06.702049 29321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.695852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.130:39991 (local address 127.8.204.129:58923)
0927 22:58:06.696498 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:06.696513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:06.696622 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:06.696953 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:06.696958 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:06.696983 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:58:06.697287 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.697294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.698440 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.698447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.699972 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:06.699985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.701160 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.701167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:06.701319 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.701366 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:58:06.701472 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 22:58:06.701620 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":62,"thread_start_us":326,"threads_started":1}
I20260927 22:58:06.703130 29248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b722fcc6d2d43c6a98a7ca18cdf3193" candidate_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69259272343f4221b590cae4eb5a5c00" is_pre_election: true
I20260927 22:58:06.703071 29118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b722fcc6d2d43c6a98a7ca18cdf3193" candidate_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1be4d2154e149db9da40e4368806c5e" is_pre_election: true
I20260927 22:58:06.703712 29118 raft_consensus.cc:2454] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135fe6a6543b4fa2a0ca1545ea3ab9fb in term 0.
I20260927 22:58:06.703712 29248 raft_consensus.cc:2454] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 135fe6a6543b4fa2a0ca1545ea3ab9fb in term 0.
I20260927 22:58:06.704416 28923 leader_election.cc:302] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135fe6a6543b4fa2a0ca1545ea3ab9fb, e1be4d2154e149db9da40e4368806c5e; no voters:
I20260927 22:58:06.704748 29319 raft_consensus.cc:2781] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:58:06.704895 29319 raft_consensus.cc:482] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:58:06.705041 29319 raft_consensus.cc:3037] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:06.707139 29319 raft_consensus.cc:504] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.707762 29319 leader_election.cc:288] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [CANDIDATE]: Term 1 election: Requested vote from peers 69259272343f4221b590cae4eb5a5c00 (127.8.204.131:36283), e1be4d2154e149db9da40e4368806c5e (127.8.204.130:39991)
I20260927 22:58:06.708252 29248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b722fcc6d2d43c6a98a7ca18cdf3193" candidate_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69259272343f4221b590cae4eb5a5c00"
I20260927 22:58:06.708278 29118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b722fcc6d2d43c6a98a7ca18cdf3193" candidate_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1be4d2154e149db9da40e4368806c5e"
I20260927 22:58:06.708468 29118 raft_consensus.cc:3037] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:06.708468 29248 raft_consensus.cc:3037] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:06.711666 29118 raft_consensus.cc:2454] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135fe6a6543b4fa2a0ca1545ea3ab9fb in term 1.
I20260927 22:58:06.711664 29248 raft_consensus.cc:2454] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 135fe6a6543b4fa2a0ca1545ea3ab9fb in term 1.
I20260927 22:58:06.712047 28923 leader_election.cc:302] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135fe6a6543b4fa2a0ca1545ea3ab9fb, e1be4d2154e149db9da40e4368806c5e; no voters:
I20260927 22:58:06.712293 29319 raft_consensus.cc:2781] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:06.712706 29319 raft_consensus.cc:686] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 1 LEADER]: Becoming Leader. State: Replica: 135fe6a6543b4fa2a0ca1545ea3ab9fb, State: Running, Role: LEADER
I20260927 22:58:06.713150 29319 consensus_queue.cc:237] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } }
I20260927 22:58:06.717331 28845 catalog_manager.cc:5495] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 135fe6a6543b4fa2a0ca1545ea3ab9fb (127.8.204.129). New cstate: current_term: 1 leader_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "135fe6a6543b4fa2a0ca1545ea3ab9fb" member_type: VOTER last_known_addr { host: "127.8.204.129" port: 36637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 } health_report { overall_health: UNKNOWN } } }
I20260927 22:58:06.779999 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.773702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.8.204.129:36637)
0927 22:58:06.773970 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.773977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.773999 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:58:06.774108 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.774114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.774242 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.774412 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:06.774838 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.775994 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.777885 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.778301 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.778406 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.778803 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:06.779350 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:06.779461 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.779545 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.779619 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:58:06.788975 29118 raft_consensus.cc:1261] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 1 FOLLOWER]: Refusing update from remote peer 135fe6a6543b4fa2a0ca1545ea3ab9fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:06.789027 29248 raft_consensus.cc:1261] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 1 FOLLOWER]: Refusing update from remote peer 135fe6a6543b4fa2a0ca1545ea3ab9fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:06.789908 29322 consensus_queue.cc:1035] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1be4d2154e149db9da40e4368806c5e" member_type: VOTER last_known_addr { host: "127.8.204.130" port: 39991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:06.790253 29319 consensus_queue.cc:1035] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "69259272343f4221b590cae4eb5a5c00" member_type: VOTER last_known_addr { host: "127.8.204.131" port: 36283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:06.801687 29325 mvcc.cc:205] Tried to move back new op lower bound from 7334092336277057536 to 7334092335980531712. Current Snapshot: MvccSnapshot[applied={T|T < 7334092336277057536}]
I20260927 22:58:06.815882 28844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:58:06.816200 28844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:58:06.820148 28844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b2ef5334814e98a88eb12c429d3988: Sending DeleteTablet for 3 replicas of tablet 0b722fcc6d2d43c6a98a7ca18cdf3193
I20260927 22:58:06.821396 28967 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:58:06 UTC) from {username='slave'} at 127.0.0.1:45982
I20260927 22:58:06.822347 29098 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:58:06 UTC) from {username='slave'} at 127.0.0.1:53060
I20260927 22:58:06.822819 29335 tablet_replica.cc:330] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: stopping tablet replica
I20260927 22:58:06.821880 29228 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:58:06 UTC) from {username='slave'} at 127.0.0.1:51602
I20260927 22:58:06.823328 29335 raft_consensus.cc:2229] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:58:06.824419 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28905
I20260927 22:58:06.824707 29335 raft_consensus.cc:2258] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:58:06.834012 29337 tablet_replica.cc:330] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: stopping tablet replica
I20260927 22:58:06.834586 29337 raft_consensus.cc:2229] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:58:06.834619 29335 ts_tablet_manager.cc:1905] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:58:06.835232 29337 raft_consensus.cc:2258] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:58:06.836741 29336 tablet_replica.cc:330] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: stopping tablet replica
I20260927 22:58:06.837271 29336 raft_consensus.cc:2229] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:58:06.837841 29336 raft_consensus.cc:2258] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:58:06.838177 29337 ts_tablet_manager.cc:1905] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:58:06.842900 29335 ts_tablet_manager.cc:1918] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:58:06.843170 29335 log.cc:1196] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-0/wal/wals/0b722fcc6d2d43c6a98a7ca18cdf3193
I20260927 22:58:06.843884 29335 ts_tablet_manager.cc:1939] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 135fe6a6543b4fa2a0ca1545ea3ab9fb: Deleting consensus metadata
I20260927 22:58:06.844717 29336 ts_tablet_manager.cc:1905] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:58:06.846030 28829 catalog_manager.cc:4841] TS 135fe6a6543b4fa2a0ca1545ea3ab9fb (127.8.204.129:36637): tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 (table test-table [id=c5c1f12a92064a078c585f0521c06590]) successfully deleted
I20260927 22:58:06.845659 29337 ts_tablet_manager.cc:1918] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:58:06.846437 29337 log.cc:1196] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-2/wal/wals/0b722fcc6d2d43c6a98a7ca18cdf3193
I20260927 22:58:06.846971 29337 ts_tablet_manager.cc:1939] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P 69259272343f4221b590cae4eb5a5c00: Deleting consensus metadata
I20260927 22:58:06.848860 28832 catalog_manager.cc:4841] TS 69259272343f4221b590cae4eb5a5c00 (127.8.204.131:36283): tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 (table test-table [id=c5c1f12a92064a078c585f0521c06590]) successfully deleted
I20260927 22:58:06.849555 29306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.825894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.8.204.129:36637)
0927 22:58:06.826122 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.826131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.839750 (+ 13619us) server_negotiation.cc:407] Connection header received
0927 22:58:06.840203 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.840209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.840347 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.840436 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:06.841150 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.842340 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.844506 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.845182 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.845292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.847813 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:06.847823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:06.847840 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:06.848667 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:06.848739 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:06.848764 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:06.848843 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.849095 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.849215 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:58:06.850657 29027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:06.851980 29336 ts_tablet_manager.cc:1918] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:58:06.852166 29336 log.cc:1196] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Deleting WAL directory at /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790549572925352-9010-0/minicluster-data/ts-1/wal/wals/0b722fcc6d2d43c6a98a7ca18cdf3193
I20260927 22:58:06.852706 29336 ts_tablet_manager.cc:1939] T 0b722fcc6d2d43c6a98a7ca18cdf3193 P e1be4d2154e149db9da40e4368806c5e: Deleting consensus metadata
I20260927 22:58:06.854420 28832 catalog_manager.cc:4841] TS e1be4d2154e149db9da40e4368806c5e (127.8.204.130:39991): tablet 0b722fcc6d2d43c6a98a7ca18cdf3193 (table test-table [id=c5c1f12a92064a078c585f0521c06590]) successfully deleted
I20260927 22:58:06.972339 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28905
I20260927 22:58:06.990351 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29036
I20260927 22:58:06.997110 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:06.991099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53070 (local address 127.8.204.130:39991)
0927 22:58:06.991346 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:06.991352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:06.991379 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:58:06.991508 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:06.991514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:06.991619 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:06.991706 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:06.992108 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.993002 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.994235 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:06.994646 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:06.994752 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:06.995851 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:06.995860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:06.995874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:06.996480 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:06.996526 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:06.996544 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:06.996640 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:06.996758 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 22:58:06.996847 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:58:06.997735 29158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:07.087790 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29036
I20260927 22:58:07.105259 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29166
I20260927 22:58:07.111629 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:07.105971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.8.204.131:36283)
0927 22:58:07.106146 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:07.106151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:07.106169 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:58:07.106362 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:07.106366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:07.106447 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:07.106539 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:07.106925 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:07.107718 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:07.108908 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:07.109275 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:07.109369 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:07.110356 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:07.110365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:07.110378 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:07.110984 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:07.111029 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:07.111048 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:07.111136 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:07.111250 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:58:07.111391 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:58:07.112305 29288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:07.199488 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29166
I20260927 22:58:07.215889 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 28814
I20260927 22:58:07.216943 28875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:07.325973 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 28814
2026-09-27T22:58:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2963 ms)
[----------] 6 tests from EncryptionPolicyTest (21868 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:07 dist-test-slave-40vx krb5kdc[29345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:07 dist-test-slave-40vx krb5kdc[29345](info): set up 2 sockets
Sep 27 22:58:07 dist-test-slave-40vx krb5kdc[29345](info): commencing operation
krb5kdc: starting...
W20260927 22:58:08.447753 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:08 dist-test-slave-40vx krb5kdc[29345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:08.690946 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:09.013008 29358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:09.013345 29358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:09.013523 29358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:09.020851 29358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:09.021020 29358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:09.021135 29358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:09.021239 29358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:09.021323 29358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:09.031456 29358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:09.032792 29358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:09.034132 29358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:09.045758 29364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:09.045768 29366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:09.045914 29363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:09.046689 29358 server_base.cc:1034] running on GCE node
I20260927 22:58:09.047534 29358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:09.048951 29358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:09.050153 29358 hybrid_clock.cc:648] HybridClock initialized: now 1790549889050109 us; error 34 us; skew 500 ppm
I20260927 22:58:09.050518 29358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:09 dist-test-slave-40vx krb5kdc[29345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549889, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:09.057076 29358 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:09.058617 29358 webserver.cc:460] Webserver started at http://127.0.0.1:37861/ using document root <none> and password file <none>
I20260927 22:58:09.059386 29358 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:09.059540 29358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:09.059841 29358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:09.062737 29358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "105dc8074ae645b99faf4c1ad6c1936f"
format_stamp: "Formatted at 2026-09-27 22:58:09 on dist-test-slave-40vx"
server_key: "a99bb7afb93e2ed80f1f707bd75d9410"
server_key_iv: "18a8fc07f9b0ffd4ed85052f6d39f4c0"
server_key_version: "encryptionkey@0"
I20260927 22:58:09.063535 29358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "105dc8074ae645b99faf4c1ad6c1936f"
format_stamp: "Formatted at 2026-09-27 22:58:09 on dist-test-slave-40vx"
server_key: "a99bb7afb93e2ed80f1f707bd75d9410"
server_key_iv: "18a8fc07f9b0ffd4ed85052f6d39f4c0"
server_key_version: "encryptionkey@0"
I20260927 22:58:09.068382 29358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:58:09.071668 29373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:09.072898 29358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:58:09.073074 29358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "105dc8074ae645b99faf4c1ad6c1936f"
format_stamp: "Formatted at 2026-09-27 22:58:09 on dist-test-slave-40vx"
server_key: "a99bb7afb93e2ed80f1f707bd75d9410"
server_key_iv: "18a8fc07f9b0ffd4ed85052f6d39f4c0"
server_key_version: "encryptionkey@0"
I20260927 22:58:09.073256 29358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:09.106992 29358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:09.111289 29358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:09.111595 29358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:09.129525 29358 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43859
I20260927 22:58:09.129544 29424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43859 every 8 connection(s)
I20260927 22:58:09.130944 29358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:09.134003 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29358
I20260927 22:58:09.134202 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:09.134982 9010 external_mini_cluster.cc:1428] Setting key 83b19d85931404f225355a51fd77be3a
I20260927 22:58:09.135161 29425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:09.149478 29425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f: Bootstrap starting.
Sep 27 22:58:09 dist-test-slave-40vx krb5kdc[29345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:58:09.153137 29425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:09.154179 29425 log.cc:824] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:09.156838 29425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f: No bootstrap required, opened a new log
I20260927 22:58:09.158471 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:09.136909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59739 (local address 127.0.0.1:43859)
0927 22:58:09.137685 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:09.137699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:09.137743 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:58:09.140131 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:09.140159 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:09.140731 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:09.141185 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:09.142527 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:09.144436 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:09.145672 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:09.146750 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:09.146942 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:09.151239 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:09.151247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:09.151259 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:09.151344 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:09.154343 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:09.155248 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:09.155254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:09.155259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:09.155487 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:09.155941 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:09.155946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:09.155950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:09.156454 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:09.156763 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:09.157225 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 22:58:09.157541 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":320,"threads_started":1}
I20260927 22:58:09.162515 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:09.165215 29425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105dc8074ae645b99faf4c1ad6c1936f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260927 22:58:09.165563 29425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:09.165665 29425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105dc8074ae645b99faf4c1ad6c1936f, State: Initialized, Role: FOLLOWER
I20260927 22:58:09.166290 29425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105dc8074ae645b99faf4c1ad6c1936f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260927 22:58:09.166525 29425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:09.166649 29425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:09.166815 29425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:09.170149 29425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105dc8074ae645b99faf4c1ad6c1936f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260927 22:58:09.170709 29425 leader_election.cc:302] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 105dc8074ae645b99faf4c1ad6c1936f; no voters:
I20260927 22:58:09.171361 29425 leader_election.cc:288] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:09.171492 29435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:09.172722 29435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [term 1 LEADER]: Becoming Leader. State: Replica: 105dc8074ae645b99faf4c1ad6c1936f, State: Running, Role: LEADER
I20260927 22:58:09.173511 29425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:09.173401 29435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105dc8074ae645b99faf4c1ad6c1936f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } }
I20260927 22:58:09.180135 29437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 105dc8074ae645b99faf4c1ad6c1936f. Latest consensus state: current_term: 1 leader_uuid: "105dc8074ae645b99faf4c1ad6c1936f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105dc8074ae645b99faf4c1ad6c1936f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } } }
I20260927 22:58:09.179512 29436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "105dc8074ae645b99faf4c1ad6c1936f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "105dc8074ae645b99faf4c1ad6c1936f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43859 } } }
I20260927 22:58:09.180675 29437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:09.180788 29436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:09.188766 29428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:09.167518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55455 (local address 127.0.0.1:43859)
0927 22:58:09.167744 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:09.167753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:09.168181 (+ 428us) server_negotiation.cc:407] Connection header received
0927 22:58:09.168330 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:09.168334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:09.168399 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:09.168481 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:09.170334 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:09.171264 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:09.173025 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:09.173811 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:09.173950 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:09.177547 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:09.177558 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:09.177565 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:09.177618 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:09.183240 (+ 5622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:09.184608 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:09.184617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:09.184623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:09.184806 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:09.185512 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:09.185520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:09.185526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:09.186015 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:09.186360 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:09.187988 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0927 22:58:09.188234 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:58:09.193921 29448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:09.198882 29448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:09.206848 29448 catalog_manager.cc:1347] Generated new cluster ID: 1b3076f93d3e45c1ad3c5094ad0d1ae0
I20260927 22:58:09.206961 29448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:09.224681 29448 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:09.226536 29448 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:09.235417 29448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 105dc8074ae645b99faf4c1ad6c1936f: Generated new TSK 0
I20260927 22:58:09.236433 29448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:09.242915 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29358
I20260927 22:58:09.244158 29420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:09.335702 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29358
W20260927 22:58:09.352705 29343 connection.cc:535] client connection to 127.0.0.1:43859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43859): Connection reset by peer (error 104)
2026-09-27T22:58:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:09 dist-test-slave-40vx krb5kdc[29457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:09 dist-test-slave-40vx krb5kdc[29457](info): set up 2 sockets
Sep 27 22:58:09 dist-test-slave-40vx krb5kdc[29457](info): commencing operation
krb5kdc: starting...
W20260927 22:58:10.435657 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:10 dist-test-slave-40vx krb5kdc[29457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:10.678731 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:11.004379 29470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:11.004657 29470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:11.004848 29470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:11.011991 29470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:11.012094 29470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:11.012184 29470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:11.012267 29470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:11.012353 29470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:11.022246 29470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:11.023625 29470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:11.024873 29470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:11.036918 29475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:11.037029 29476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:11.036984 29478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:11.038026 29470 server_base.cc:1034] running on GCE node
I20260927 22:58:11.038755 29470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:11.040256 29470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:11.041481 29470 hybrid_clock.cc:648] HybridClock initialized: now 1790549891041441 us; error 36 us; skew 500 ppm
I20260927 22:58:11.041813 29470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:11 dist-test-slave-40vx krb5kdc[29457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:11.044853 29470 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:11.046320 29470 webserver.cc:460] Webserver started at http://127.0.0.1:34021/ using document root <none> and password file <none>
I20260927 22:58:11.046993 29470 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:11.047137 29470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:11.047505 29470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:11.050383 29470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "df9bd807a2104df6b14672dc8ad98c0d"
format_stamp: "Formatted at 2026-09-27 22:58:11 on dist-test-slave-40vx"
server_key: "f123e02b6591f296f1678d153c5eff9b"
server_key_iv: "4b4a108541c73a7555fa42ada4937a2e"
server_key_version: "encryptionkey@0"
I20260927 22:58:11.051337 29470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "df9bd807a2104df6b14672dc8ad98c0d"
format_stamp: "Formatted at 2026-09-27 22:58:11 on dist-test-slave-40vx"
server_key: "f123e02b6591f296f1678d153c5eff9b"
server_key_iv: "4b4a108541c73a7555fa42ada4937a2e"
server_key_version: "encryptionkey@0"
I20260927 22:58:11.056385 29470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:58:11.059734 29485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:11.061223 29470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 22:58:11.061404 29470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "df9bd807a2104df6b14672dc8ad98c0d"
format_stamp: "Formatted at 2026-09-27 22:58:11 on dist-test-slave-40vx"
server_key: "f123e02b6591f296f1678d153c5eff9b"
server_key_iv: "4b4a108541c73a7555fa42ada4937a2e"
server_key_version: "encryptionkey@0"
I20260927 22:58:11.061605 29470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:11.099426 29470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:11.103449 29470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:11.103729 29470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:11.121961 29470 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43937
I20260927 22:58:11.121986 29536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43937 every 8 connection(s)
I20260927 22:58:11.123395 29470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:11.127036 29537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:11.132539 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29470
I20260927 22:58:11.132745 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:11.133169 9010 external_mini_cluster.cc:1428] Setting key db09ca014fbbd8bcdb4da73f1674d5b1
I20260927 22:58:11.138587 29537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d: Bootstrap starting.
I20260927 22:58:11.142395 29537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:11.143555 29537 log.cc:824] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:11.146216 29537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d: No bootstrap required, opened a new log
I20260927 22:58:11.147953 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:11.134981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45157 (local address 127.0.0.1:43937)
0927 22:58:11.135888 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:11.135904 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:11.135956 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:58:11.138307 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:11.138333 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:11.138829 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:11.139248 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:11.139772 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:11.141769 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:11.143157 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:11.144293 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:11.144498 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:11.145703 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:11.145710 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:11.145723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:11.146225 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:11.146296 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:11.146351 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:11.146531 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:11.146804 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 22:58:11.146968 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":431,"threads_started":1}
I20260927 22:58:11.151798 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:11.154520 29537 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9bd807a2104df6b14672dc8ad98c0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 22:58:11.154891 29537 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:11.155004 29537 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9bd807a2104df6b14672dc8ad98c0d, State: Initialized, Role: FOLLOWER
I20260927 22:58:11.155624 29537 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9bd807a2104df6b14672dc8ad98c0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 22:58:11.155870 29537 raft_consensus.cc:388] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:11.156018 29537 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:11.156220 29537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:11.159219 29537 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9bd807a2104df6b14672dc8ad98c0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 22:58:11.159783 29537 leader_election.cc:302] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df9bd807a2104df6b14672dc8ad98c0d; no voters:
I20260927 22:58:11.160426 29537 leader_election.cc:288] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:11.160544 29547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:11.162156 29547 raft_consensus.cc:686] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [term 1 LEADER]: Becoming Leader. State: Replica: df9bd807a2104df6b14672dc8ad98c0d, State: Running, Role: LEADER
I20260927 22:58:11.162652 29537 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:11.162715 29547 consensus_queue.cc:237] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9bd807a2104df6b14672dc8ad98c0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } }
I20260927 22:58:11.163528 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:11.156135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33281 (local address 127.0.0.1:43937)
0927 22:58:11.156292 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:11.156299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:11.156864 (+ 565us) server_negotiation.cc:407] Connection header received
0927 22:58:11.157012 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:11.157017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:11.157097 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:11.157203 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:11.157604 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:11.158728 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:11.160161 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:11.160995 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:11.161134 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:11.162605 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:11.162612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:11.162616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:11.162640 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:11.162686 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:11.162714 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:11.162837 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:11.163108 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 22:58:11.163165 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 22:58:11.166744 29548 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df9bd807a2104df6b14672dc8ad98c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9bd807a2104df6b14672dc8ad98c0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } } }
I20260927 22:58:11.167093 29548 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:11.167559 29549 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader df9bd807a2104df6b14672dc8ad98c0d. Latest consensus state: current_term: 1 leader_uuid: "df9bd807a2104df6b14672dc8ad98c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df9bd807a2104df6b14672dc8ad98c0d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43937 } } }
I20260927 22:58:11.168606 29549 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:11.175197 29555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:11.180650 29555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:11.189446 29555 catalog_manager.cc:1347] Generated new cluster ID: 93a8be07e9bc44c0956b061259e0c932
I20260927 22:58:11.189586 29555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:11.219671 29555 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:11.221604 29555 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:11.229835 29555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df9bd807a2104df6b14672dc8ad98c0d: Generated new TSK 0
I20260927 22:58:11.230963 29555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:11.249482 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29470
W20260927 22:58:11.250864 29532 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45157
W20260927 22:58:11.251600 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29470: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:11.251770 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29470
W20260927 22:58:11.265756 29455 connection.cc:535] client connection to 127.0.0.1:43937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43937): Connection reset by peer (error 104)
2026-09-27T22:58:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:11 dist-test-slave-40vx krb5kdc[29569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:11 dist-test-slave-40vx krb5kdc[29569](info): set up 2 sockets
Sep 27 22:58:11 dist-test-slave-40vx krb5kdc[29569](info): commencing operation
krb5kdc: starting...
W20260927 22:58:12.367800 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:12 dist-test-slave-40vx krb5kdc[29569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:12.612794 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:12.937218 29582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:12.937552 29582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:12.937704 29582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:12.944780 29582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:12.944936 29582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:12.945057 29582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:12.945150 29582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:12.945217 29582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:12.954506 29582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:12.955744 29582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:12.957065 29582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:12.968565 29590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:12.968627 29587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:12.968534 29588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:12.969393 29582 server_base.cc:1034] running on GCE node
I20260927 22:58:12.970161 29582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:12.971527 29582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:12.972764 29582 hybrid_clock.cc:648] HybridClock initialized: now 1790549892972730 us; error 44 us; skew 500 ppm
I20260927 22:58:12.973140 29582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:12 dist-test-slave-40vx krb5kdc[29569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:12.976259 29582 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:12.978000 29582 webserver.cc:460] Webserver started at http://127.0.0.1:38901/ using document root <none> and password file <none>
I20260927 22:58:12.978711 29582 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:12.978868 29582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:12.979176 29582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:12.982225 29582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "36e02045708e4180a1fb0478fbc271e7"
format_stamp: "Formatted at 2026-09-27 22:58:12 on dist-test-slave-40vx"
server_key: "e598e0597be61afaaf7690b655bdfb51"
server_key_iv: "792df8fc5452ccf81095259fc4241517"
server_key_version: "encryptionkey@0"
I20260927 22:58:12.983031 29582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "36e02045708e4180a1fb0478fbc271e7"
format_stamp: "Formatted at 2026-09-27 22:58:12 on dist-test-slave-40vx"
server_key: "e598e0597be61afaaf7690b655bdfb51"
server_key_iv: "792df8fc5452ccf81095259fc4241517"
server_key_version: "encryptionkey@0"
I20260927 22:58:12.988484 29582 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:58:12.992178 29597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:12.993625 29582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:12.993817 29582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "36e02045708e4180a1fb0478fbc271e7"
format_stamp: "Formatted at 2026-09-27 22:58:12 on dist-test-slave-40vx"
server_key: "e598e0597be61afaaf7690b655bdfb51"
server_key_iv: "792df8fc5452ccf81095259fc4241517"
server_key_version: "encryptionkey@0"
I20260927 22:58:12.994030 29582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:12.999818 29582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:13.003854 29582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:13.004138 29582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:13.022954 29582 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34871
I20260927 22:58:13.022990 29648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34871 every 8 connection(s)
I20260927 22:58:13.024360 29582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:13.025812 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29582
I20260927 22:58:13.026005 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:13.026443 9010 external_mini_cluster.cc:1428] Setting key cfb2ca7351cc30d0855cba9c7f97d17b
I20260927 22:58:13.028789 29649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:13.034003 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:13.028218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53601 (local address 127.0.0.1:34871)
0927 22:58:13.029030 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:13.029042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:13.029084 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:58:13.030655 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:13.030674 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:13.031015 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:13.031452 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:13.032454 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:13.032460 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:13.032471 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:13.032484 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:13.032557 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:13.032579 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:13.032810 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:13.032953 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:58:13.033126 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":317,"threads_started":1}
I20260927 22:58:13.037499 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:13.039767 29649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7: Bootstrap starting.
I20260927 22:58:13.042927 29649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:13.044283 29649 log.cc:824] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:13.044344 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:13.041919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50951 (local address 127.0.0.1:34871)
0927 22:58:13.042138 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:13.042143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:13.042411 (+ 268us) server_negotiation.cc:407] Connection header received
0927 22:58:13.042564 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:13.042568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:13.042611 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:13.042687 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:13.043698 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:13.043702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:13.043705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:13.043719 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:13.043757 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:13.043771 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:13.043854 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:13.044003 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 22:58:13.044046 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:58:13.046993 29649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7: No bootstrap required, opened a new log
I20260927 22:58:13.053928 29649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e02045708e4180a1fb0478fbc271e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260927 22:58:13.054279 29649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:13.054396 29649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e02045708e4180a1fb0478fbc271e7, State: Initialized, Role: FOLLOWER
I20260927 22:58:13.055104 29649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e02045708e4180a1fb0478fbc271e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260927 22:58:13.055434 29649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:13.055572 29649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:13.055755 29649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:13.058041 29649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e02045708e4180a1fb0478fbc271e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260927 22:58:13.058542 29649 leader_election.cc:302] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36e02045708e4180a1fb0478fbc271e7; no voters:
I20260927 22:58:13.059108 29649 leader_election.cc:288] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:13.059259 29659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:13.059985 29659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [term 1 LEADER]: Becoming Leader. State: Replica: 36e02045708e4180a1fb0478fbc271e7, State: Running, Role: LEADER
I20260927 22:58:13.060563 29659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e02045708e4180a1fb0478fbc271e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } }
I20260927 22:58:13.061316 29649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:13.063870 29660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36e02045708e4180a1fb0478fbc271e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e02045708e4180a1fb0478fbc271e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } } }
I20260927 22:58:13.063971 29661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36e02045708e4180a1fb0478fbc271e7. Latest consensus state: current_term: 1 leader_uuid: "36e02045708e4180a1fb0478fbc271e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e02045708e4180a1fb0478fbc271e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34871 } } }
I20260927 22:58:13.064302 29660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:13.064363 29661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:13.066354 29666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:13.072475 29666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:13.080554 29666 catalog_manager.cc:1347] Generated new cluster ID: 33e763cae9864269b8d669a66ba7e268
I20260927 22:58:13.080660 29666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:13.094238 29666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:13.094666 29666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:13.110186 29666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36e02045708e4180a1fb0478fbc271e7: Generated new TSK 0
I20260927 22:58:13.111119 29666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:13.116595 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29582
W20260927 22:58:13.117606 29644 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53601
W20260927 22:58:13.118239 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:13.118382 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29582
2026-09-27T22:58:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:13 dist-test-slave-40vx krb5kdc[29680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:13 dist-test-slave-40vx krb5kdc[29680](info): set up 2 sockets
Sep 27 22:58:13 dist-test-slave-40vx krb5kdc[29680](info): commencing operation
krb5kdc: starting...
W20260927 22:58:14.215740 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:14 dist-test-slave-40vx krb5kdc[29680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:14.467993 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:14.802852 29693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:14.803222 29693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:14.803431 29693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:14.812240 29693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:14.812388 29693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:14.812497 29693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:14.812605 29693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:14.812691 29693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:14.822942 29693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:14.824379 29693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:14.825940 29693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:14.838223 29698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:14.838269 29701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:14.838317 29699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:14.839172 29693 server_base.cc:1034] running on GCE node
I20260927 22:58:14.839984 29693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:14.841480 29693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:14.842679 29693 hybrid_clock.cc:648] HybridClock initialized: now 1790549894842634 us; error 42 us; skew 500 ppm
I20260927 22:58:14.843055 29693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:14 dist-test-slave-40vx krb5kdc[29680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:14.851037 29693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:14.852754 29693 webserver.cc:460] Webserver started at http://127.0.0.1:33101/ using document root <none> and password file <none>
I20260927 22:58:14.853461 29693 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:14.853561 29693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:14.853820 29693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:14.856909 29693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "092f85c170ba45efbcc51dfa2133fa95"
format_stamp: "Formatted at 2026-09-27 22:58:14 on dist-test-slave-40vx"
server_key: "05bfd794f0908ce8781b6f10d2fe82f9"
server_key_iv: "939144069eca5493d5f8415ec9588e78"
server_key_version: "encryptionkey@0"
I20260927 22:58:14.857646 29693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "092f85c170ba45efbcc51dfa2133fa95"
format_stamp: "Formatted at 2026-09-27 22:58:14 on dist-test-slave-40vx"
server_key: "05bfd794f0908ce8781b6f10d2fe82f9"
server_key_iv: "939144069eca5493d5f8415ec9588e78"
server_key_version: "encryptionkey@0"
I20260927 22:58:14.862869 29693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:58:14.866335 29708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:14.867761 29693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:14.867954 29693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "092f85c170ba45efbcc51dfa2133fa95"
format_stamp: "Formatted at 2026-09-27 22:58:14 on dist-test-slave-40vx"
server_key: "05bfd794f0908ce8781b6f10d2fe82f9"
server_key_iv: "939144069eca5493d5f8415ec9588e78"
server_key_version: "encryptionkey@0"
I20260927 22:58:14.868145 29693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:14.883874 29693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:14.888039 29693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:14.888334 29693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:14.906962 29693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39497
I20260927 22:58:14.906997 29759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39497 every 8 connection(s)
I20260927 22:58:14.908357 29693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:14.912038 29760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:14.912142 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29693
I20260927 22:58:14.912339 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:14.912777 9010 external_mini_cluster.cc:1428] Setting key 2f95fdbedabaa6c25231453af8d4a8d3
I20260927 22:58:14.925473 29760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95: Bootstrap starting.
Sep 27 22:58:14 dist-test-slave-40vx krb5kdc[29680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:58:14.929014 29760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:14.930173 29760 log.cc:824] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:14.932988 29760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95: No bootstrap required, opened a new log
I20260927 22:58:14.934770 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:14.914533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51709 (local address 127.0.0.1:39497)
0927 22:58:14.915370 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:14.915387 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:14.915442 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:58:14.917152 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:14.917181 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:14.917725 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:14.918185 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:14.919462 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:14.920744 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:14.922073 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:14.922775 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:14.923000 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:14.927101 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:14.927109 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:14.927123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:14.927169 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:14.930565 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:14.931350 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:14.931357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:14.931364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:14.931539 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:14.932010 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:14.932016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:14.932019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:14.932560 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:14.932877 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:14.933331 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:58:14.933781 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":334,"threads_started":1}
I20260927 22:58:14.938483 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:14.941066 29760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f85c170ba45efbcc51dfa2133fa95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260927 22:58:14.941550 29760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:14.941682 29760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092f85c170ba45efbcc51dfa2133fa95, State: Initialized, Role: FOLLOWER
I20260927 22:58:14.942268 29760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f85c170ba45efbcc51dfa2133fa95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260927 22:58:14.942533 29760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:14.942694 29760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:14.942894 29760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:14.945319 29760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f85c170ba45efbcc51dfa2133fa95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260927 22:58:14.945849 29760 leader_election.cc:302] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092f85c170ba45efbcc51dfa2133fa95; no voters:
I20260927 22:58:14.946506 29760 leader_election.cc:288] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:14.946611 29770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:14.948210 29770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [term 1 LEADER]: Becoming Leader. State: Replica: 092f85c170ba45efbcc51dfa2133fa95, State: Running, Role: LEADER
I20260927 22:58:14.948832 29770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f85c170ba45efbcc51dfa2133fa95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } }
I20260927 22:58:14.948990 29760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:14.953698 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092f85c170ba45efbcc51dfa2133fa95. Latest consensus state: current_term: 1 leader_uuid: "092f85c170ba45efbcc51dfa2133fa95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f85c170ba45efbcc51dfa2133fa95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } } }
I20260927 22:58:14.955345 29771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092f85c170ba45efbcc51dfa2133fa95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f85c170ba45efbcc51dfa2133fa95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39497 } } }
I20260927 22:58:14.955687 29771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:14.956058 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:14.956550 29781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:14.959439 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:14.942805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58583 (local address 127.0.0.1:39497)
0927 22:58:14.942994 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:14.943001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:14.943732 (+ 731us) server_negotiation.cc:407] Connection header received
0927 22:58:14.943929 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:14.943934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:14.944015 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:14.944127 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:14.945519 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:14.946288 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:14.948008 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:14.948715 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:14.948851 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:14.951903 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:14.951912 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:14.951922 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:14.951976 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:14.955757 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:14.956649 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:14.956655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:14.956658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:14.956814 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:14.957330 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:14.957339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:14.957345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:14.957874 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:14.958257 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:14.958621 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 22:58:14.958933 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:58:14.962643 29781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:14.971591 29781 catalog_manager.cc:1347] Generated new cluster ID: ef55150b488f453d9db1c006181535af
I20260927 22:58:14.971750 29781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:14.981081 29781 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:14.982415 29781 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:14.991463 29781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 092f85c170ba45efbcc51dfa2133fa95: Generated new TSK 0
I20260927 22:58:14.992290 29781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:14.999444 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29693
I20260927 22:58:15.000602 29755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:15.091722 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29693
2026-09-27T22:58:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:15 dist-test-slave-40vx krb5kdc[29792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:15 dist-test-slave-40vx krb5kdc[29792](info): set up 2 sockets
Sep 27 22:58:15 dist-test-slave-40vx krb5kdc[29792](info): commencing operation
krb5kdc: starting...
W20260927 22:58:16.187721 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:16 dist-test-slave-40vx krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:16.441908 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:16.767946 29805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:16.768226 29805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:16.768420 29805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:16.775641 29805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:16.775753 29805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:16.775866 29805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:16.775974 29805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:16.776060 29805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:16.785820 29805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:16.787184 29805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:16.788573 29805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:16.800329 29810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:16.800371 29811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:16.800420 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:16.801703 29805 server_base.cc:1034] running on GCE node
I20260927 22:58:16.802523 29805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:16.804076 29805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:16.805310 29805 hybrid_clock.cc:648] HybridClock initialized: now 1790549896805268 us; error 38 us; skew 500 ppm
I20260927 22:58:16.805722 29805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:16 dist-test-slave-40vx krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:16.808825 29805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:16.810374 29805 webserver.cc:460] Webserver started at http://127.0.0.1:43821/ using document root <none> and password file <none>
I20260927 22:58:16.811067 29805 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:16.811211 29805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:16.811547 29805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:16.814692 29805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "78241098b4414e1e955f44051eb5d248"
format_stamp: "Formatted at 2026-09-27 22:58:16 on dist-test-slave-40vx"
server_key: "687eb582bbc9ff343f849fc224465339"
server_key_iv: "6c2dd05745559e1b25c84def03329d06"
server_key_version: "encryptionkey@0"
I20260927 22:58:16.815778 29805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "78241098b4414e1e955f44051eb5d248"
format_stamp: "Formatted at 2026-09-27 22:58:16 on dist-test-slave-40vx"
server_key: "687eb582bbc9ff343f849fc224465339"
server_key_iv: "6c2dd05745559e1b25c84def03329d06"
server_key_version: "encryptionkey@0"
I20260927 22:58:16.821686 29805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:58:16.825137 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:16.826406 29805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:58:16.826591 29805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "78241098b4414e1e955f44051eb5d248"
format_stamp: "Formatted at 2026-09-27 22:58:16 on dist-test-slave-40vx"
server_key: "687eb582bbc9ff343f849fc224465339"
server_key_iv: "6c2dd05745559e1b25c84def03329d06"
server_key_version: "encryptionkey@0"
I20260927 22:58:16.826793 29805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:16.846555 29805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:16.850899 29805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:16.851197 29805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:16.869946 29805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35617
I20260927 22:58:16.869971 29871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35617 every 8 connection(s)
I20260927 22:58:16.871389 29805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:16.875002 29872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:16.875499 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29805
I20260927 22:58:16.875669 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:16.876217 9010 external_mini_cluster.cc:1428] Setting key 42549fa891e3d51e15aeb5e80e6c7913
I20260927 22:58:16.887920 29872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248: Bootstrap starting.
I20260927 22:58:16.890373 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:16.878070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47415 (local address 127.0.0.1:35617)
0927 22:58:16.878941 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:16.878952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:16.879003 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:58:16.881540 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:16.881566 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:16.882037 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:16.882485 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:16.883049 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:16.884424 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:16.885782 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:16.886466 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:16.886659 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:16.887987 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:16.887995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:16.888041 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:16.888630 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:16.888709 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:16.888734 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:16.888862 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:16.889114 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 22:58:16.889405 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":359,"threads_started":1}
I20260927 22:58:16.891258 29872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:16.892372 29872 log.cc:824] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:16.894106 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:16.895774 29872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248: No bootstrap required, opened a new log
I20260927 22:58:16.906013 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:16.899335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38083 (local address 127.0.0.1:35617)
0927 22:58:16.899532 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:16.899538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:16.899913 (+ 375us) server_negotiation.cc:407] Connection header received
0927 22:58:16.900056 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:16.900062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:16.900147 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:16.900271 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:16.900741 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:16.901896 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:16.903219 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:16.903967 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:16.904124 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:16.905253 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:16.905259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:16.905264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:16.905286 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:16.905333 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:16.905361 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:16.905453 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:16.905588 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 22:58:16.905685 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:58:16.905979 29872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78241098b4414e1e955f44051eb5d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260927 22:58:16.906533 29872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:16.906661 29872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78241098b4414e1e955f44051eb5d248, State: Initialized, Role: FOLLOWER
I20260927 22:58:16.907380 29872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78241098b4414e1e955f44051eb5d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260927 22:58:16.907647 29872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:16.907792 29872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:16.907986 29872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:16.910190 29872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78241098b4414e1e955f44051eb5d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260927 22:58:16.910737 29872 leader_election.cc:302] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78241098b4414e1e955f44051eb5d248; no voters:
I20260927 22:58:16.911383 29872 leader_election.cc:288] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:16.911542 29882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:16.912217 29882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [term 1 LEADER]: Becoming Leader. State: Replica: 78241098b4414e1e955f44051eb5d248, State: Running, Role: LEADER
I20260927 22:58:16.912806 29882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78241098b4414e1e955f44051eb5d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } }
I20260927 22:58:16.913551 29872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:16.916381 29883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78241098b4414e1e955f44051eb5d248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78241098b4414e1e955f44051eb5d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } } }
I20260927 22:58:16.916117 29884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78241098b4414e1e955f44051eb5d248. Latest consensus state: current_term: 1 leader_uuid: "78241098b4414e1e955f44051eb5d248" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78241098b4414e1e955f44051eb5d248" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35617 } } }
I20260927 22:58:16.917013 29883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:16.917053 29884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:16.920274 29891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:16.925432 29891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:16.933907 29891 catalog_manager.cc:1347] Generated new cluster ID: 09f597a42979435d85cf57584d6276f7
I20260927 22:58:16.934024 29891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:16.962314 29891 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:16.963799 29891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:16.969897 29891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78241098b4414e1e955f44051eb5d248: Generated new TSK 0
I20260927 22:58:16.970825 29891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:16.976197 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29805
W20260927 22:58:16.977363 29867 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47415
W20260927 22:58:16.978255 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29805: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:16.978432 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29805
2026-09-27T22:58:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1883 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:17 dist-test-slave-40vx krb5kdc[29903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:17 dist-test-slave-40vx krb5kdc[29903](info): set up 2 sockets
Sep 27 22:58:17 dist-test-slave-40vx krb5kdc[29903](info): commencing operation
krb5kdc: starting...
W20260927 22:58:18.079725 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:18 dist-test-slave-40vx krb5kdc[29903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:18.324059 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:18.653654 29916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:18.653981 29916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:18.654150 29916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:18.661466 29916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:18.661580 29916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:18.661691 29916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:18.661788 29916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:18.661856 29916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:18.672063 29916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:18.673390 29916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:18.674710 29916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:18.686673 29924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:18.686632 29921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:18.686731 29922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:18.687640 29916 server_base.cc:1034] running on GCE node
I20260927 22:58:18.688439 29916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:18.689926 29916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:18.691175 29916 hybrid_clock.cc:648] HybridClock initialized: now 1790549898691130 us; error 40 us; skew 500 ppm
I20260927 22:58:18.691565 29916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:18 dist-test-slave-40vx krb5kdc[29903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:18.694969 29916 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:18.696817 29916 webserver.cc:460] Webserver started at http://127.0.0.1:44827/ using document root <none> and password file <none>
I20260927 22:58:18.697553 29916 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:18.697703 29916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:18.698021 29916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:18.701104 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "c194deff2d15465d9868bea079a4f024"
format_stamp: "Formatted at 2026-09-27 22:58:18 on dist-test-slave-40vx"
server_key: "dcd0b8efbd3736397447d83a818467e4"
server_key_iv: "a4b0d04701d4d10c99d68c8970b72501"
server_key_version: "encryptionkey@0"
I20260927 22:58:18.701871 29916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "c194deff2d15465d9868bea079a4f024"
format_stamp: "Formatted at 2026-09-27 22:58:18 on dist-test-slave-40vx"
server_key: "dcd0b8efbd3736397447d83a818467e4"
server_key_iv: "a4b0d04701d4d10c99d68c8970b72501"
server_key_version: "encryptionkey@0"
I20260927 22:58:18.707185 29916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260927 22:58:18.710772 29931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:18.712383 29916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:18.712574 29916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "c194deff2d15465d9868bea079a4f024"
format_stamp: "Formatted at 2026-09-27 22:58:18 on dist-test-slave-40vx"
server_key: "dcd0b8efbd3736397447d83a818467e4"
server_key_iv: "a4b0d04701d4d10c99d68c8970b72501"
server_key_version: "encryptionkey@0"
I20260927 22:58:18.712778 29916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:18.718643 29916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:18.723385 29916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:18.723685 29916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:18.741801 29916 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34507
I20260927 22:58:18.741844 29982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34507 every 8 connection(s)
I20260927 22:58:18.743148 29916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:18.746667 29983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:18.747648 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 29916
I20260927 22:58:18.747831 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:18.748427 9010 external_mini_cluster.cc:1428] Setting key f6fa92c5971d1c135e6df210abae4dce
I20260927 22:58:18.755950 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:18.750159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37897 (local address 127.0.0.1:34507)
0927 22:58:18.750923 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:18.750934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:18.750973 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:58:18.752822 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:18.752843 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:18.753183 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:18.753494 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:18.754465 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:18.754471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:18.754483 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:18.754496 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:18.754551 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:18.754572 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:18.754725 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:18.754848 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 22:58:18.755057 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":358,"threads_started":1}
I20260927 22:58:18.759374 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:18.759392 29983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024: Bootstrap starting.
I20260927 22:58:18.762974 29983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:18.764295 29983 log.cc:824] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:18.766714 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:18.764392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53011 (local address 127.0.0.1:34507)
0927 22:58:18.764606 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:18.764611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:18.764955 (+ 344us) server_negotiation.cc:407] Connection header received
0927 22:58:18.765096 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:18.765103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:18.765167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:18.765280 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:18.766075 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:18.766079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:18.766082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:18.766095 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:18.766131 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:18.766144 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:18.766226 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:18.766361 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 22:58:18.766404 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:58:18.767581 29983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024: No bootstrap required, opened a new log
I20260927 22:58:18.776348 29983 raft_consensus.cc:348] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194deff2d15465d9868bea079a4f024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260927 22:58:18.776827 29983 raft_consensus.cc:374] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:18.776921 29983 raft_consensus.cc:729] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c194deff2d15465d9868bea079a4f024, State: Initialized, Role: FOLLOWER
I20260927 22:58:18.777454 29983 consensus_queue.cc:260] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194deff2d15465d9868bea079a4f024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260927 22:58:18.777663 29983 raft_consensus.cc:388] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:18.777767 29983 raft_consensus.cc:482] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:18.777902 29983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:18.780259 29983 raft_consensus.cc:504] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194deff2d15465d9868bea079a4f024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260927 22:58:18.780725 29983 leader_election.cc:302] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c194deff2d15465d9868bea079a4f024; no voters:
I20260927 22:58:18.781255 29983 leader_election.cc:288] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:18.781407 29993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:18.782087 29993 raft_consensus.cc:686] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [term 1 LEADER]: Becoming Leader. State: Replica: c194deff2d15465d9868bea079a4f024, State: Running, Role: LEADER
I20260927 22:58:18.782595 29993 consensus_queue.cc:237] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194deff2d15465d9868bea079a4f024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } }
I20260927 22:58:18.783577 29983 sys_catalog.cc:564] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:18.785591 29995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c194deff2d15465d9868bea079a4f024. Latest consensus state: current_term: 1 leader_uuid: "c194deff2d15465d9868bea079a4f024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194deff2d15465d9868bea079a4f024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } } }
I20260927 22:58:18.785583 29994 sys_catalog.cc:455] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c194deff2d15465d9868bea079a4f024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c194deff2d15465d9868bea079a4f024" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34507 } } }
I20260927 22:58:18.786021 29995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:18.786274 29994 sys_catalog.cc:458] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:18.788681 29999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:18.793766 29999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:18.802983 29999 catalog_manager.cc:1347] Generated new cluster ID: f95a8cf288de4589a01cae0e78ee6963
I20260927 22:58:18.803108 29999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:18.847842 29999 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:18.848366 29999 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:18.863686 29999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c194deff2d15465d9868bea079a4f024: Generated new TSK 0
I20260927 22:58:18.864638 29999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:18.873369 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29916
W20260927 22:58:18.874383 29978 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37897
W20260927 22:58:18.875015 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 29916: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:18.875145 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 29916
2026-09-27T22:58:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1899 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:18 dist-test-slave-40vx krb5kdc[30014](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:18 dist-test-slave-40vx krb5kdc[30014](info): set up 2 sockets
Sep 27 22:58:18 dist-test-slave-40vx krb5kdc[30014](info): commencing operation
krb5kdc: starting...
W20260927 22:58:19.971876 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:20 dist-test-slave-40vx krb5kdc[30014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:20.223563 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42045
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:20.551180 30027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:20.551537 30027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:20.551710 30027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:20.558962 30027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:20.559119 30027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:20.559235 30027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:20.559363 30027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:20.559458 30027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:20.569104 30027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:20.570478 30027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:20.571843 30027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:20.583681 30032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:20.583765 30033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:20.583793 30035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:20.584765 30027 server_base.cc:1034] running on GCE node
I20260927 22:58:20.585564 30027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:20.586954 30027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:20.588148 30027 hybrid_clock.cc:648] HybridClock initialized: now 1790549900588102 us; error 37 us; skew 500 ppm
I20260927 22:58:20.588510 30027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:20 dist-test-slave-40vx krb5kdc[30014](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:20.595662 30027 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:20.597136 30027 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260927 22:58:20.597774 30027 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:20.597868 30027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:20.598124 30027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:20.601060 30027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "5b83804a7c334463afd9ebd11f6dffec"
format_stamp: "Formatted at 2026-09-27 22:58:20 on dist-test-slave-40vx"
server_key: "506c3b23e73948eb12c071f5088e1a9b"
server_key_iv: "4b600e02e193d2f178da7547a3fb07d9"
server_key_version: "encryptionkey@0"
I20260927 22:58:20.601727 30027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "5b83804a7c334463afd9ebd11f6dffec"
format_stamp: "Formatted at 2026-09-27 22:58:20 on dist-test-slave-40vx"
server_key: "506c3b23e73948eb12c071f5088e1a9b"
server_key_iv: "4b600e02e193d2f178da7547a3fb07d9"
server_key_version: "encryptionkey@0"
I20260927 22:58:20.606866 30027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 22:58:20.610265 30042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:20.611636 30027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:20.611778 30027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "5b83804a7c334463afd9ebd11f6dffec"
format_stamp: "Formatted at 2026-09-27 22:58:20 on dist-test-slave-40vx"
server_key: "506c3b23e73948eb12c071f5088e1a9b"
server_key_iv: "4b600e02e193d2f178da7547a3fb07d9"
server_key_version: "encryptionkey@0"
I20260927 22:58:20.611931 30027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:20.627851 30027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:20.631731 30027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:20.631958 30027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:20.650182 30027 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42045
I20260927 22:58:20.650207 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42045 every 8 connection(s)
I20260927 22:58:20.651626 30027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:20.655459 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:20.657886 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30027
I20260927 22:58:20.658075 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:20.658591 9010 external_mini_cluster.cc:1428] Setting key 7a461109cd1362c138ea5bdf22a430b1
I20260927 22:58:20.667811 30094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec: Bootstrap starting.
I20260927 22:58:20.671861 30094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec: Neither blocks nor log segments found. Creating new log.
Sep 27 22:58:20 dist-test-slave-40vx krb5kdc[30014](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:58:20.673200 30094 log.cc:824] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:20.677106 30094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec: No bootstrap required, opened a new log
I20260927 22:58:20.683736 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:20.660393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:33957 (local address 127.0.0.1:42045)
0927 22:58:20.661196 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:20.661207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:20.661247 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:58:20.662868 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:20.662889 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:20.663323 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:20.663660 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:20.665302 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:20.666613 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:20.668427 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:20.669083 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:20.669224 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:20.674556 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:20.674565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:20.674629 (+ 64us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:20.674678 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:20.678714 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:20.679702 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:20.679712 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:20.679720 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:20.679950 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:20.680399 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:20.680409 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:20.680412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:20.681158 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:20.681503 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:20.681920 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 22:58:20.682261 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":363,"threads_started":1}
I20260927 22:58:20.687000 30094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b83804a7c334463afd9ebd11f6dffec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42045 } }
I20260927 22:58:20.687487 30094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:20.687636 30094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b83804a7c334463afd9ebd11f6dffec, State: Initialized, Role: FOLLOWER
I20260927 22:58:20.687898 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:20.688354 30094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b83804a7c334463afd9ebd11f6dffec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42045 } }
I20260927 22:58:20.688633 30094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:20.688766 30094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:20.688930 30094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:20.692499 30094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b83804a7c334463afd9ebd11f6dffec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42045 } }
I20260927 22:58:20.693173 30094 leader_election.cc:302] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b83804a7c334463afd9ebd11f6dffec; no voters:
I20260927 22:58:20.693890 30094 leader_election.cc:288] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:20.694013 30104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:20.695569 30104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [term 1 LEADER]: Becoming Leader. State: Replica: 5b83804a7c334463afd9ebd11f6dffec, State: Running, Role: LEADER
I20260927 22:58:20.696307 30094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:20.696278 30104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b83804a7c334463afd9ebd11f6dffec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42045 } }
I20260927 22:58:20.702960 30106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b83804a7c334463afd9ebd11f6dffec. Latest consensus state: current_term: 1 leader_uuid: "5b83804a7c334463afd9ebd11f6dffec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b83804a7c334463afd9ebd11f6dffec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42045 } } }
I20260927 22:58:20.703379 30106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:20.703951 30105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b83804a7c334463afd9ebd11f6dffec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b83804a7c334463afd9ebd11f6dffec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42045 } } }
I20260927 22:58:20.704272 30105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:20.715373 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:20.693184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:59307 (local address 127.0.0.1:42045)
0927 22:58:20.693374 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:20.693380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:20.693957 (+ 577us) server_negotiation.cc:407] Connection header received
0927 22:58:20.694170 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:20.694176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:20.694280 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:20.694400 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:20.696523 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:20.697729 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:20.700143 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:20.701855 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:20.701998 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:20.705561 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:20.705571 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:20.705579 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:20.705630 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:20.710663 (+ 5033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:20.712017 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:20.712025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:20.712029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:20.712180 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:20.712810 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:20.712817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:20.712823 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:20.713340 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:20.714556 (+ 1216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:20.714751 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 22:58:20.714970 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:58:20.715741 30120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:58:20.715873 30120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:58:20.716054 30113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:20.717744 30113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:20.725765 30113 catalog_manager.cc:1347] Generated new cluster ID: 613a1512d0a84a589f865feb2d2f3a2c
I20260927 22:58:20.725915 30113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:20.760322 30113 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:20.762159 30113 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:20.772810 30113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b83804a7c334463afd9ebd11f6dffec: Generated new TSK 0
I20260927 22:58:20.773643 30113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:20.784612 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30027
I20260927 22:58:20.786126 30089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:20.882126 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30027
W20260927 22:58:20.899498 30012 connection.cc:535] client connection to 127.0.0.1:42045 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42045): Connection reset by peer (error 104)
2026-09-27T22:58:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2007 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:20 dist-test-slave-40vx krb5kdc[30126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:20 dist-test-slave-40vx krb5kdc[30126](info): set up 2 sockets
Sep 27 22:58:20 dist-test-slave-40vx krb5kdc[30126](info): commencing operation
krb5kdc: starting...
W20260927 22:58:21.987836 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:22 dist-test-slave-40vx krb5kdc[30126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:22.250914 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:22.579017 30139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:22.579346 30139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:22.579527 30139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:22.586803 30139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:22.586908 30139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:22.586993 30139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:22.587103 30139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:22.587190 30139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:22.597113 30139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:22.598394 30139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:22.599804 30139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:22.611830 30144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:22.611860 30147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:22.611790 30145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:22.612674 30139 server_base.cc:1034] running on GCE node
I20260927 22:58:22.613508 30139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:22.614857 30139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:22.616111 30139 hybrid_clock.cc:648] HybridClock initialized: now 1790549902616064 us; error 53 us; skew 500 ppm
I20260927 22:58:22.616496 30139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:22 dist-test-slave-40vx krb5kdc[30126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:22.619657 30139 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:22.621142 30139 webserver.cc:460] Webserver started at http://127.0.0.1:40185/ using document root <none> and password file <none>
I20260927 22:58:22.621833 30139 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:22.621984 30139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:22.622285 30139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:22.625283 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "aa7921b18d2342c8b336cf6f448fcafa"
format_stamp: "Formatted at 2026-09-27 22:58:22 on dist-test-slave-40vx"
server_key: "c5c9911d0f626b033b6f482bcc9ed8b1"
server_key_iv: "b4190087cff018c35a3569eaa0621ef7"
server_key_version: "encryptionkey@0"
I20260927 22:58:22.626048 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "aa7921b18d2342c8b336cf6f448fcafa"
format_stamp: "Formatted at 2026-09-27 22:58:22 on dist-test-slave-40vx"
server_key: "c5c9911d0f626b033b6f482bcc9ed8b1"
server_key_iv: "b4190087cff018c35a3569eaa0621ef7"
server_key_version: "encryptionkey@0"
I20260927 22:58:22.631036 30139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:58:22.634480 30154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:22.635927 30139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:22.636103 30139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "aa7921b18d2342c8b336cf6f448fcafa"
format_stamp: "Formatted at 2026-09-27 22:58:22 on dist-test-slave-40vx"
server_key: "c5c9911d0f626b033b6f482bcc9ed8b1"
server_key_iv: "b4190087cff018c35a3569eaa0621ef7"
server_key_version: "encryptionkey@0"
I20260927 22:58:22.636304 30139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:22.656298 30139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:22.660899 30139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:22.661201 30139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:22.680711 30139 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38647
I20260927 22:58:22.680732 30205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38647 every 8 connection(s)
I20260927 22:58:22.682175 30139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:22.685194 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30139
I20260927 22:58:22.685405 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:22.685853 9010 external_mini_cluster.cc:1428] Setting key efe3bb372548412911456201e6b4f29b
I20260927 22:58:22.686053 30206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:22.700938 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:22.687703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:49055 (local address 127.0.0.1:38647)
0927 22:58:22.688474 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:22.688490 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:22.688540 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:58:22.690909 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:22.690936 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:22.691491 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:22.691938 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:22.692474 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:22.694482 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:22.695833 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:22.696859 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:22.697055 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:22.698213 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:22.698224 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:22.698247 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:22.698956 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:22.699066 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:22.699100 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:22.699293 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:22.699567 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:58:22.699721 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":346,"threads_started":1}
I20260927 22:58:22.701608 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa: Bootstrap starting.
I20260927 22:58:22.704963 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:22.705211 30206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:22.706415 30206 log.cc:824] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:22.709762 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa: No bootstrap required, opened a new log
I20260927 22:58:22.716661 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:22.710301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:46169 (local address 127.0.0.1:38647)
0927 22:58:22.710903 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:22.710910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:22.710929 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:58:22.711183 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:22.711187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:22.711248 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:22.711397 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:22.711939 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:22.712737 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:22.714069 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:22.714548 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:22.714737 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:22.715904 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:22.715909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:22.715913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:22.715930 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:22.715979 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:22.715994 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:22.716135 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:22.716314 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 22:58:22.716373 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
I20260927 22:58:22.718391 30206 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7921b18d2342c8b336cf6f448fcafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38647 } }
I20260927 22:58:22.718815 30206 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:22.718914 30206 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7921b18d2342c8b336cf6f448fcafa, State: Initialized, Role: FOLLOWER
I20260927 22:58:22.719614 30206 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7921b18d2342c8b336cf6f448fcafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38647 } }
I20260927 22:58:22.719901 30206 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:22.720058 30206 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:22.720263 30206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:22.722576 30206 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7921b18d2342c8b336cf6f448fcafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38647 } }
I20260927 22:58:22.723032 30206 leader_election.cc:302] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa7921b18d2342c8b336cf6f448fcafa; no voters:
I20260927 22:58:22.723616 30206 leader_election.cc:288] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:22.723770 30216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:22.724457 30216 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [term 1 LEADER]: Becoming Leader. State: Replica: aa7921b18d2342c8b336cf6f448fcafa, State: Running, Role: LEADER
I20260927 22:58:22.725059 30216 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7921b18d2342c8b336cf6f448fcafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38647 } }
I20260927 22:58:22.725780 30206 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:22.727638 30217 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa7921b18d2342c8b336cf6f448fcafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7921b18d2342c8b336cf6f448fcafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38647 } } }
I20260927 22:58:22.728039 30217 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:22.729753 30218 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa7921b18d2342c8b336cf6f448fcafa. Latest consensus state: current_term: 1 leader_uuid: "aa7921b18d2342c8b336cf6f448fcafa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa7921b18d2342c8b336cf6f448fcafa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38647 } } }
I20260927 22:58:22.730137 30218 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:22.731599 30221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:22.736780 30221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:22.745713 30221 catalog_manager.cc:1347] Generated new cluster ID: 42b0d62903fc46fcbf3a032a8a4908b9
I20260927 22:58:22.745831 30221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:22.756402 30221 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:22.758205 30221 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:22.768234 30221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa7921b18d2342c8b336cf6f448fcafa: Generated new TSK 0
I20260927 22:58:22.769366 30221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:22.785080 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30139
W20260927 22:58:22.786811 30201 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.214:49055
W20260927 22:58:22.787595 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30139: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:22.787730 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30139
W20260927 22:58:22.801497 30124 connection.cc:535] client connection to 127.0.0.1:38647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38647): Connection reset by peer (error 104)
2026-09-27T22:58:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:22 dist-test-slave-40vx krb5kdc[30237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:22 dist-test-slave-40vx krb5kdc[30237](info): set up 2 sockets
Sep 27 22:58:22 dist-test-slave-40vx krb5kdc[30237](info): commencing operation
krb5kdc: starting...
W20260927 22:58:23.887853 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:24 dist-test-slave-40vx krb5kdc[30237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:24.139235 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:24.466641 30250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:24.466908 30250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:24.466997 30250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:24.473976 30250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:24.474079 30250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:24.474126 30250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:24.474196 30250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:24.474244 30250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:24.483747 30250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:24.484930 30250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:24.486171 30250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:24.498096 30258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:24.498207 30255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:24.498232 30256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:24.499099 30250 server_base.cc:1034] running on GCE node
I20260927 22:58:24.499935 30250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:24.501183 30250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:24.502405 30250 hybrid_clock.cc:648] HybridClock initialized: now 1790549904502374 us; error 46 us; skew 500 ppm
I20260927 22:58:24.502743 30250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:24 dist-test-slave-40vx krb5kdc[30237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:24.505805 30250 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:24.507232 30250 webserver.cc:460] Webserver started at http://127.0.0.1:42915/ using document root <none> and password file <none>
I20260927 22:58:24.507970 30250 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:24.508116 30250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:24.508419 30250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:24.511222 30250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "a5f9059cb8c54b6f8689e13e16712002"
format_stamp: "Formatted at 2026-09-27 22:58:24 on dist-test-slave-40vx"
server_key: "c31f774510bf2d8fb59911af28ef5ccb"
server_key_iv: "75a39a4fb586165e5cbec99d31c58dbe"
server_key_version: "encryptionkey@0"
I20260927 22:58:24.512086 30250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "a5f9059cb8c54b6f8689e13e16712002"
format_stamp: "Formatted at 2026-09-27 22:58:24 on dist-test-slave-40vx"
server_key: "c31f774510bf2d8fb59911af28ef5ccb"
server_key_iv: "75a39a4fb586165e5cbec99d31c58dbe"
server_key_version: "encryptionkey@0"
I20260927 22:58:24.516997 30250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:58:24.520298 30265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:24.522056 30250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:58:24.522215 30250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "a5f9059cb8c54b6f8689e13e16712002"
format_stamp: "Formatted at 2026-09-27 22:58:24 on dist-test-slave-40vx"
server_key: "c31f774510bf2d8fb59911af28ef5ccb"
server_key_iv: "75a39a4fb586165e5cbec99d31c58dbe"
server_key_version: "encryptionkey@0"
I20260927 22:58:24.522498 30250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:24.530390 30250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:24.534286 30250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:24.534571 30250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:24.552721 30250 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34701
I20260927 22:58:24.552776 30316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34701 every 8 connection(s)
I20260927 22:58:24.554059 30250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:24.557680 30317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:24.564301 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30250
I20260927 22:58:24.564509 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:24.565063 9010 external_mini_cluster.cc:1428] Setting key e9355d6f3a9507a59fb33b8502c576e1
I20260927 22:58:24.569263 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002: Bootstrap starting.
I20260927 22:58:24.572618 30317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:24.572924 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:24.567006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:56535 (local address 127.0.0.1:34701)
0927 22:58:24.567799 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:24.567810 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:24.567849 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:58:24.569575 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:24.569597 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:24.569921 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:24.570307 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:24.571362 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:24.571368 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:24.571381 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:24.571395 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:24.571450 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:24.571469 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:24.571686 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:24.571812 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 22:58:24.571993 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":357,"threads_started":1}
I20260927 22:58:24.573745 30317 log.cc:824] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:24.576853 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:24.576889 30317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002: No bootstrap required, opened a new log
I20260927 22:58:24.585669 30320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:24.581021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:35849 (local address 127.0.0.1:34701)
0927 22:58:24.581222 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:24.581226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:24.583772 (+ 2546us) server_negotiation.cc:407] Connection header received
0927 22:58:24.583965 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:24.583969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:24.584011 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:24.584140 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:24.585046 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:24.585058 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:24.585062 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:24.585080 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:24.585118 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:24.585132 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:24.585261 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:24.585369 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 22:58:24.585424 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:58:24.586262 30317 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9059cb8c54b6f8689e13e16712002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34701 } }
I20260927 22:58:24.586643 30317 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:24.586773 30317 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f9059cb8c54b6f8689e13e16712002, State: Initialized, Role: FOLLOWER
I20260927 22:58:24.587368 30317 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9059cb8c54b6f8689e13e16712002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34701 } }
I20260927 22:58:24.587582 30317 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:24.587723 30317 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:24.587898 30317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:24.589953 30317 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9059cb8c54b6f8689e13e16712002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34701 } }
I20260927 22:58:24.590372 30317 leader_election.cc:302] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5f9059cb8c54b6f8689e13e16712002; no voters:
I20260927 22:58:24.590826 30317 leader_election.cc:288] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:24.590969 30327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:24.591727 30327 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [term 1 LEADER]: Becoming Leader. State: Replica: a5f9059cb8c54b6f8689e13e16712002, State: Running, Role: LEADER
I20260927 22:58:24.592289 30327 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9059cb8c54b6f8689e13e16712002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34701 } }
I20260927 22:58:24.592918 30317 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:24.595221 30328 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5f9059cb8c54b6f8689e13e16712002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9059cb8c54b6f8689e13e16712002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34701 } } }
I20260927 22:58:24.595345 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5f9059cb8c54b6f8689e13e16712002. Latest consensus state: current_term: 1 leader_uuid: "a5f9059cb8c54b6f8689e13e16712002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5f9059cb8c54b6f8689e13e16712002" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34701 } } }
I20260927 22:58:24.595825 30328 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:24.595875 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:24.598161 30336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:24.603185 30336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:24.612241 30336 catalog_manager.cc:1347] Generated new cluster ID: d8415d8f8baf4319b38a61a5f39eeb43
I20260927 22:58:24.612376 30336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:24.632098 30336 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:24.632522 30336 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:24.641803 30336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5f9059cb8c54b6f8689e13e16712002: Generated new TSK 0
I20260927 22:58:24.642632 30336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:24.648847 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30250
W20260927 22:58:24.650071 30312 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.214:56535
W20260927 22:58:24.650730 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30250: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:24.650869 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30250
2026-09-27T22:58:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1862 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:24 dist-test-slave-40vx krb5kdc[30348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:24 dist-test-slave-40vx krb5kdc[30348](info): set up 2 sockets
Sep 27 22:58:24 dist-test-slave-40vx krb5kdc[30348](info): commencing operation
krb5kdc: starting...
W20260927 22:58:25.747859 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:25 dist-test-slave-40vx krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:26.002349 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:26.325089 30361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:26.325367 30361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:26.325457 30361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:26.332397 30361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:26.332508 30361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:26.332554 30361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:26.332614 30361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:26.332657 30361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:26.342867 30361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:26.344165 30361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:26.345515 30361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:26.357157 30366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:26.357340 30367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:26.357131 30369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:26.358459 30361 server_base.cc:1034] running on GCE node
I20260927 22:58:26.359261 30361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:26.360714 30361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:26.361912 30361 hybrid_clock.cc:648] HybridClock initialized: now 1790549906361878 us; error 25 us; skew 500 ppm
I20260927 22:58:26.362218 30361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:26 dist-test-slave-40vx krb5kdc[30348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:26.370873 30361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:26.372563 30361 webserver.cc:460] Webserver started at http://127.0.0.1:37789/ using document root <none> and password file <none>
I20260927 22:58:26.373243 30361 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:26.373343 30361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:26.373579 30361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:26.376644 30361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "881c2eff6a95432fa9900c357d02a6b7"
format_stamp: "Formatted at 2026-09-27 22:58:26 on dist-test-slave-40vx"
server_key: "294d3179f08520a37f470691dcbd2beb"
server_key_iv: "169e0c0e7435ea4d9d91c1456b96ca0b"
server_key_version: "encryptionkey@0"
I20260927 22:58:26.377363 30361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "881c2eff6a95432fa9900c357d02a6b7"
format_stamp: "Formatted at 2026-09-27 22:58:26 on dist-test-slave-40vx"
server_key: "294d3179f08520a37f470691dcbd2beb"
server_key_iv: "169e0c0e7435ea4d9d91c1456b96ca0b"
server_key_version: "encryptionkey@0"
I20260927 22:58:26.382670 30361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:58:26.386292 30376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:26.387861 30361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 22:58:26.388075 30361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "881c2eff6a95432fa9900c357d02a6b7"
format_stamp: "Formatted at 2026-09-27 22:58:26 on dist-test-slave-40vx"
server_key: "294d3179f08520a37f470691dcbd2beb"
server_key_iv: "169e0c0e7435ea4d9d91c1456b96ca0b"
server_key_version: "encryptionkey@0"
I20260927 22:58:26.388250 30361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:26.418813 30361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:26.423650 30361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:26.423964 30361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:26.442943 30361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39911
I20260927 22:58:26.442984 30427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39911 every 8 connection(s)
I20260927 22:58:26.444360 30361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:26.446714 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30361
I20260927 22:58:26.446897 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:26.447443 9010 external_mini_cluster.cc:1428] Setting key 03671b53daaf0a89556d2cbbf69701c1
I20260927 22:58:26.448845 30428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:58:26 dist-test-slave-40vx krb5kdc[30348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:58:26.463611 30428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7: Bootstrap starting.
I20260927 22:58:26.467942 30428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:26.469205 30428 log.cc:824] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:26.472754 30428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7: No bootstrap required, opened a new log
I20260927 22:58:26.472802 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:26.449423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:60267 (local address 127.0.0.1:39911)
0927 22:58:26.450175 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:26.450186 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:26.450225 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:58:26.451834 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:26.451855 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:26.452267 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:26.452604 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:26.454318 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:26.455767 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:26.457669 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:26.458453 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:26.458602 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:26.463796 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:26.463807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:26.463828 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:26.463875 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:26.467941 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:26.468878 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:26.468887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:26.468897 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:26.469111 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:26.469627 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:26.469631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:26.469634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:26.470289 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:26.470535 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:26.471136 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 22:58:26.471492 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":334,"threads_started":1}
I20260927 22:58:26.476887 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:26.481688 30428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881c2eff6a95432fa9900c357d02a6b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39911 } }
I20260927 22:58:26.482120 30428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:26.482254 30428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881c2eff6a95432fa9900c357d02a6b7, State: Initialized, Role: FOLLOWER
I20260927 22:58:26.482895 30428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881c2eff6a95432fa9900c357d02a6b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39911 } }
I20260927 22:58:26.483158 30428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:26.483349 30428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:26.483541 30428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:26.485776 30428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881c2eff6a95432fa9900c357d02a6b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39911 } }
I20260927 22:58:26.486326 30428 leader_election.cc:302] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881c2eff6a95432fa9900c357d02a6b7; no voters:
I20260927 22:58:26.487013 30428 leader_election.cc:288] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:26.487118 30438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:26.489714 30428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:26.489936 30438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [term 1 LEADER]: Becoming Leader. State: Replica: 881c2eff6a95432fa9900c357d02a6b7, State: Running, Role: LEADER
I20260927 22:58:26.490471 30438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881c2eff6a95432fa9900c357d02a6b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39911 } }
I20260927 22:58:26.494333 30440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881c2eff6a95432fa9900c357d02a6b7. Latest consensus state: current_term: 1 leader_uuid: "881c2eff6a95432fa9900c357d02a6b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881c2eff6a95432fa9900c357d02a6b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39911 } } }
I20260927 22:58:26.494733 30440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:26.499123 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:26.481192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:56365 (local address 127.0.0.1:39911)
0927 22:58:26.481379 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:26.481387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:26.481749 (+ 362us) server_negotiation.cc:407] Connection header received
0927 22:58:26.481983 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:26.481986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:26.482074 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:26.482205 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:26.483913 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:26.484726 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:26.485972 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:26.486445 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:26.486576 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:26.489292 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:26.489304 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:26.489308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:26.489336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:26.494697 (+ 5361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:26.496084 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:26.496092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:26.496098 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:26.496331 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:26.496988 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:26.496996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:26.497002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:26.497478 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:26.497848 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:26.498329 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 22:58:26.498581 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 22:58:26.499866 30439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881c2eff6a95432fa9900c357d02a6b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881c2eff6a95432fa9900c357d02a6b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39911 } } }
I20260927 22:58:26.500181 30439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:26.503994 30446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:26.510633 30446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:26.519019 30446 catalog_manager.cc:1347] Generated new cluster ID: 8e44b0ae98b0425b8b8a5a69d89bff28
I20260927 22:58:26.519140 30446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:26.530527 30446 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:26.532435 30446 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:26.546693 30446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881c2eff6a95432fa9900c357d02a6b7: Generated new TSK 0
I20260927 22:58:26.547659 30446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:26.556752 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30361
I20260927 22:58:26.558056 30423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:26.649758 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30361
W20260927 22:58:26.666630 30346 connection.cc:535] client connection to 127.0.0.1:39911 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39911): Connection reset by peer (error 104)
2026-09-27T22:58:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:26 dist-test-slave-40vx krb5kdc[30460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:26 dist-test-slave-40vx krb5kdc[30460](info): set up 2 sockets
Sep 27 22:58:26 dist-test-slave-40vx krb5kdc[30460](info): commencing operation
krb5kdc: starting...
W20260927 22:58:27.747885 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:27 dist-test-slave-40vx krb5kdc[30460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:28.007083 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:28.330103 30473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:28.330441 30473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:28.330614 30473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:28.337841 30473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:28.338019 30473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:28.338127 30473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:28.338238 30473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:28.338325 30473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:28.348143 30473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:28.349455 30473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:28.350772 30473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:28.362619 30478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:28.362577 30481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:28.362643 30479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:28.363036 30473 server_base.cc:1034] running on GCE node
I20260927 22:58:28.363945 30473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:28.365391 30473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:28.366586 30473 hybrid_clock.cc:648] HybridClock initialized: now 1790549908366546 us; error 31 us; skew 500 ppm
I20260927 22:58:28.366950 30473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:28 dist-test-slave-40vx krb5kdc[30460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:28.370069 30473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:28.371567 30473 webserver.cc:460] Webserver started at http://127.0.0.1:37345/ using document root <none> and password file <none>
I20260927 22:58:28.372252 30473 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:28.372395 30473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:28.372716 30473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:28.375756 30473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "5c9451172ed34410b27eb8bffc9c0aaf"
format_stamp: "Formatted at 2026-09-27 22:58:28 on dist-test-slave-40vx"
server_key: "3df0ad91633f4a04506c52e70952deaa"
server_key_iv: "22975bfa5d567283200752bdad897acc"
server_key_version: "encryptionkey@0"
I20260927 22:58:28.376533 30473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "5c9451172ed34410b27eb8bffc9c0aaf"
format_stamp: "Formatted at 2026-09-27 22:58:28 on dist-test-slave-40vx"
server_key: "3df0ad91633f4a04506c52e70952deaa"
server_key_iv: "22975bfa5d567283200752bdad897acc"
server_key_version: "encryptionkey@0"
I20260927 22:58:28.381736 30473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:58:28.385108 30488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:28.386394 30473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 22:58:28.386564 30473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "5c9451172ed34410b27eb8bffc9c0aaf"
format_stamp: "Formatted at 2026-09-27 22:58:28 on dist-test-slave-40vx"
server_key: "3df0ad91633f4a04506c52e70952deaa"
server_key_iv: "22975bfa5d567283200752bdad897acc"
server_key_version: "encryptionkey@0"
I20260927 22:58:28.386758 30473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:28.409317 30473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:28.413319 30473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:28.413619 30473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:28.432561 30473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36475
I20260927 22:58:28.432593 30539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36475 every 8 connection(s)
I20260927 22:58:28.433966 30473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:28.437717 30540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:28.441504 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30473
I20260927 22:58:28.441674 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:28.442109 9010 external_mini_cluster.cc:1428] Setting key 17da87bb4915602e7a4678cd2378f480
I20260927 22:58:28.449746 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf: Bootstrap starting.
I20260927 22:58:28.452963 30540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:28.454108 30540 log.cc:824] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:28.454843 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:28.443926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:41763 (local address 127.0.0.1:36475)
0927 22:58:28.444631 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:28.444644 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:28.444684 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:58:28.446338 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:28.446360 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:28.446759 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:28.447140 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:28.447707 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:28.448967 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:28.450356 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:28.451037 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:28.451250 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:28.452464 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:28.452471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:28.452484 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:28.452992 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:28.453072 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:28.453117 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:28.453296 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:28.453578 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 22:58:28.453726 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":284,"threads_started":1}
I20260927 22:58:28.457226 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf: No bootstrap required, opened a new log
I20260927 22:58:28.458802 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:28.465832 30540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260927 22:58:28.466306 30540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:28.466413 30540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9451172ed34410b27eb8bffc9c0aaf, State: Initialized, Role: FOLLOWER
I20260927 22:58:28.467048 30540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260927 22:58:28.467360 30540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:28.467581 30540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:28.467777 30540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:28.470005 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:28.463217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:56021 (local address 127.0.0.1:36475)
0927 22:58:28.463457 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:28.463463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:28.463785 (+ 322us) server_negotiation.cc:407] Connection header received
0927 22:58:28.463948 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:28.463951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:28.464011 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:28.464098 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:28.464783 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:28.465503 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:28.467366 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:28.467963 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:28.468073 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:28.469228 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:28.469232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:28.469239 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:28.469255 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:28.469305 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:28.469319 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:28.469404 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:28.469602 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 22:58:28.469645 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 22:58:28.470909 30540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260927 22:58:28.471480 30540 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9451172ed34410b27eb8bffc9c0aaf; no voters:
I20260927 22:58:28.472041 30540 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:28.472159 30550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:28.473114 30550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [term 1 LEADER]: Becoming Leader. State: Replica: 5c9451172ed34410b27eb8bffc9c0aaf, State: Running, Role: LEADER
I20260927 22:58:28.473753 30550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260927 22:58:28.474226 30540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:28.476823 30551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } } }
I20260927 22:58:28.477209 30551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:28.476861 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9451172ed34410b27eb8bffc9c0aaf. Latest consensus state: current_term: 1 leader_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9451172ed34410b27eb8bffc9c0aaf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } } }
I20260927 22:58:28.477460 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:28.479166 30558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:28.486841 30558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:28.496105 30558 catalog_manager.cc:1347] Generated new cluster ID: deff9f04b27940f296dd7c026e84fa60
I20260927 22:58:28.496274 30558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:28.529299 30558 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:28.530619 30558 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:28.541107 30558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9451172ed34410b27eb8bffc9c0aaf: Generated new TSK 0
I20260927 22:58:28.541990 30558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:28.558030 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30473
W20260927 22:58:28.559198 30535 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.214:41763
W20260927 22:58:28.559999 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30473: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:28.560145 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30473
W20260927 22:58:28.573901 30458 connection.cc:535] client connection to 127.0.0.1:36475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36475): Connection reset by peer (error 104)
2026-09-27T22:58:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:28 dist-test-slave-40vx krb5kdc[30571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:28 dist-test-slave-40vx krb5kdc[30571](info): set up 2 sockets
Sep 27 22:58:28 dist-test-slave-40vx krb5kdc[30571](info): commencing operation
krb5kdc: starting...
W20260927 22:58:29.667915 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:29 dist-test-slave-40vx krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:58:29.926141 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:30.248186 30584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:30.248549 30584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:30.248737 30584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:30.255738 30584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:30.255895 30584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:30.255980 30584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:30.256068 30584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:30.256135 30584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:30.265676 30584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:30.267033 30584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:30.268410 30584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:30.280457 30589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:30.280427 30592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:30.280511 30590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:30.281730 30584 server_base.cc:1034] running on GCE node
I20260927 22:58:30.282523 30584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:30.284017 30584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:30.285251 30584 hybrid_clock.cc:648] HybridClock initialized: now 1790549910285215 us; error 38 us; skew 500 ppm
I20260927 22:58:30.285635 30584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:30 dist-test-slave-40vx krb5kdc[30571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:30.288861 30584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:58:30.290351 30584 webserver.cc:460] Webserver started at http://127.0.0.1:41619/ using document root <none> and password file <none>
I20260927 22:58:30.291023 30584 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:30.291167 30584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:30.291532 30584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:30.294456 30584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "3c44841db9004a17bb647975e22c74e6"
format_stamp: "Formatted at 2026-09-27 22:58:30 on dist-test-slave-40vx"
server_key: "fea04768fc5726c4ea3deccbd9b69fc7"
server_key_iv: "23f1c3c63292a9b4a7a2c6b7fff140e2"
server_key_version: "encryptionkey@0"
I20260927 22:58:30.295252 30584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "3c44841db9004a17bb647975e22c74e6"
format_stamp: "Formatted at 2026-09-27 22:58:30 on dist-test-slave-40vx"
server_key: "fea04768fc5726c4ea3deccbd9b69fc7"
server_key_iv: "23f1c3c63292a9b4a7a2c6b7fff140e2"
server_key_version: "encryptionkey@0"
I20260927 22:58:30.300581 30584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:58:30.304318 30599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:30.305688 30584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:30.305881 30584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "3c44841db9004a17bb647975e22c74e6"
format_stamp: "Formatted at 2026-09-27 22:58:30 on dist-test-slave-40vx"
server_key: "fea04768fc5726c4ea3deccbd9b69fc7"
server_key_iv: "23f1c3c63292a9b4a7a2c6b7fff140e2"
server_key_version: "encryptionkey@0"
I20260927 22:58:30.306087 30584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:30.312098 30584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:30.315938 30584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:30.316236 30584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:30.335042 30584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37015
I20260927 22:58:30.335094 30650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37015 every 8 connection(s)
I20260927 22:58:30.336473 30584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:30.339859 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30584
I20260927 22:58:30.340049 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:30.340610 9010 external_mini_cluster.cc:1428] Setting key d48a6d42d67d0ceec017c6e1f39cb5ed
I20260927 22:58:30.340654 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:30.348805 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:30.342413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:52699 (local address 127.0.0.1:37015)
0927 22:58:30.343162 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:30.343174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:30.343215 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:58:30.345367 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:30.345388 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:30.345735 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:30.346129 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:30.347164 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:30.347171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:30.347187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:30.347204 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:30.347265 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:30.347340 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:30.347575 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:30.347737 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 22:58:30.347908 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":324,"threads_started":1}
I20260927 22:58:30.352367 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:30.353947 30651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6: Bootstrap starting.
I20260927 22:58:30.357602 30651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:30.358840 30651 log.cc:824] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:30.359115 30654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:30.356644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.214:51357 (local address 127.0.0.1:37015)
0927 22:58:30.356794 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:30.356798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:30.357238 (+ 440us) server_negotiation.cc:407] Connection header received
0927 22:58:30.357426 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:30.357430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:30.357477 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:30.357583 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:30.358515 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:30.358525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:30.358528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:30.358542 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:30.358582 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:30.358597 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:30.358725 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:30.358814 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:58:30.358875 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:58:30.361742 30651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6: No bootstrap required, opened a new log
I20260927 22:58:30.368813 30651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c44841db9004a17bb647975e22c74e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37015 } }
I20260927 22:58:30.369228 30651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:30.369349 30651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c44841db9004a17bb647975e22c74e6, State: Initialized, Role: FOLLOWER
I20260927 22:58:30.369908 30651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c44841db9004a17bb647975e22c74e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37015 } }
I20260927 22:58:30.370149 30651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:30.370280 30651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:30.370440 30651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:30.372606 30651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c44841db9004a17bb647975e22c74e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37015 } }
I20260927 22:58:30.373101 30651 leader_election.cc:302] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c44841db9004a17bb647975e22c74e6; no voters:
I20260927 22:58:30.373646 30651 leader_election.cc:288] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:30.373790 30661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:30.374439 30661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c44841db9004a17bb647975e22c74e6, State: Running, Role: LEADER
I20260927 22:58:30.375005 30661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c44841db9004a17bb647975e22c74e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37015 } }
I20260927 22:58:30.375757 30651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:30.377808 30662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c44841db9004a17bb647975e22c74e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c44841db9004a17bb647975e22c74e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37015 } } }
I20260927 22:58:30.378000 30663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c44841db9004a17bb647975e22c74e6. Latest consensus state: current_term: 1 leader_uuid: "3c44841db9004a17bb647975e22c74e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c44841db9004a17bb647975e22c74e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37015 } } }
I20260927 22:58:30.378283 30663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:30.378286 30662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:30.383021 30666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:30.387984 30666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:30.396453 30666 catalog_manager.cc:1347] Generated new cluster ID: 0e5566f7705246b2af290d7121bafc8d
I20260927 22:58:30.396566 30666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:30.423468 30666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:30.423960 30666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:30.442155 30666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c44841db9004a17bb647975e22c74e6: Generated new TSK 0
I20260927 22:58:30.443013 30666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:30.455040 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30584
W20260927 22:58:30.456398 30646 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.214:52699
W20260927 22:58:30.457115 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:30.457255 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30584
2026-09-27T22:58:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:30 dist-test-slave-40vx krb5kdc[30682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:30 dist-test-slave-40vx krb5kdc[30682](info): set up 2 sockets
Sep 27 22:58:30 dist-test-slave-40vx krb5kdc[30682](info): commencing operation
krb5kdc: starting...
W20260927 22:58:31.543869 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:31 dist-test-slave-40vx krb5kdc[30682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:31.792913 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37493
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:32.126196 30695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:32.126477 30695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:32.126645 30695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:32.133971 30695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:32.134083 30695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:32.134192 30695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:32.134313 30695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:32.134378 30695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:32.144330 30695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37493
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:32.145682 30695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:32.146976 30695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:32.159403 30703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:32.159385 30701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:32.159504 30700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:32.164674 30695 server_base.cc:1034] running on GCE node
I20260927 22:58:32.165503 30695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:32.166927 30695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:32.168116 30695 hybrid_clock.cc:648] HybridClock initialized: now 1790549912168078 us; error 31 us; skew 500 ppm
I20260927 22:58:32.168473 30695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:32 dist-test-slave-40vx krb5kdc[30682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549912, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:32.175459 30695 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:32.177049 30695 webserver.cc:460] Webserver started at http://127.8.204.190:46651/ using document root <none> and password file <none>
I20260927 22:58:32.177768 30695 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:32.177896 30695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:32.178205 30695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:32.181306 30695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "d6023e2f1b5b41199eded64a7412e947"
format_stamp: "Formatted at 2026-09-27 22:58:32 on dist-test-slave-40vx"
server_key: "c54199ae5703850169b8b2f2d2f4128f"
server_key_iv: "815dd17f5248bfeaf5871f9325f6b365"
server_key_version: "encryptionkey@0"
I20260927 22:58:32.182132 30695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "d6023e2f1b5b41199eded64a7412e947"
format_stamp: "Formatted at 2026-09-27 22:58:32 on dist-test-slave-40vx"
server_key: "c54199ae5703850169b8b2f2d2f4128f"
server_key_iv: "815dd17f5248bfeaf5871f9325f6b365"
server_key_version: "encryptionkey@0"
I20260927 22:58:32.187512 30695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:58:32.190846 30710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:32.192237 30695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:32.192417 30695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "d6023e2f1b5b41199eded64a7412e947"
format_stamp: "Formatted at 2026-09-27 22:58:32 on dist-test-slave-40vx"
server_key: "c54199ae5703850169b8b2f2d2f4128f"
server_key_iv: "815dd17f5248bfeaf5871f9325f6b365"
server_key_version: "encryptionkey@0"
I20260927 22:58:32.192592 30695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:32.214059 30695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:32.221694 30695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:32.222044 30695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:32.250092 30761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:37493 every 8 connection(s)
I20260927 22:58:32.250064 30695 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:37493
I20260927 22:58:32.252247 30695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:32.257165 30762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:32.261684 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30695
I20260927 22:58:32.261876 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:32.262421 9010 external_mini_cluster.cc:1428] Setting key ef6bb3847d29af2b439298d8f8de38a5
I20260927 22:58:32.273991 30762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947: Bootstrap starting.
I20260927 22:58:32.278321 30762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:32.279788 30762 log.cc:824] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:32.283349 30762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947: No bootstrap required, opened a new log
Sep 27 22:58:32 dist-test-slave-40vx krb5kdc[30682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:32.293716 30762 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6023e2f1b5b41199eded64a7412e947" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37493 } }
I20260927 22:58:32.294198 30762 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:32.294307 30762 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6023e2f1b5b41199eded64a7412e947, State: Initialized, Role: FOLLOWER
I20260927 22:58:32.295027 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:32.264371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59879 (local address 127.8.204.190:37493)
0927 22:58:32.265869 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:32.265886 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:32.265937 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:58:32.268871 (+ 2934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:32.268902 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:32.269687 (+ 785us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:32.270211 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:32.273217 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:32.276031 (+ 2814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:32.278534 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:32.279850 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:32.280045 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:32.286168 (+ 6123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:32.286181 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:32.286208 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:32.286274 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:32.290260 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:32.291220 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:32.291226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:32.291232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:32.291527 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:32.292037 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:32.292041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:32.292043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:32.292575 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:32.292819 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:32.293310 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 22:58:32.293629 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":982,"threads_started":1}
I20260927 22:58:32.294942 30762 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6023e2f1b5b41199eded64a7412e947" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37493 } }
I20260927 22:58:32.295217 30762 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:32.295378 30762 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:32.295542 30762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:32.298499 30762 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6023e2f1b5b41199eded64a7412e947" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37493 } }
I20260927 22:58:32.299121 30762 leader_election.cc:302] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6023e2f1b5b41199eded64a7412e947; no voters:
I20260927 22:58:32.299322 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:32.299820 30762 leader_election.cc:288] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:32.299925 30767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:32.301927 30762 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:32.308296 30767 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [term 1 LEADER]: Becoming Leader. State: Replica: d6023e2f1b5b41199eded64a7412e947, State: Running, Role: LEADER
I20260927 22:58:32.308981 30767 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6023e2f1b5b41199eded64a7412e947" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37493 } }
I20260927 22:58:32.317601 30768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6023e2f1b5b41199eded64a7412e947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6023e2f1b5b41199eded64a7412e947" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37493 } } }
I20260927 22:58:32.318075 30768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:32.318683 30769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6023e2f1b5b41199eded64a7412e947. Latest consensus state: current_term: 1 leader_uuid: "d6023e2f1b5b41199eded64a7412e947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6023e2f1b5b41199eded64a7412e947" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37493 } } }
I20260927 22:58:32.319021 30769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947 [sys.catalog]: This master's current role is: LEADER
W20260927 22:58:32.332713 30786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:58:32.332866 30786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:58:32.333729 30765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:32.311126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41661 (local address 127.8.204.190:37493)
0927 22:58:32.311348 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:32.311357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:32.312491 (+ 1134us) server_negotiation.cc:407] Connection header received
0927 22:58:32.312731 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:32.312740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:32.312846 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:32.312971 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:32.315377 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:32.316689 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:32.318576 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:32.319390 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:32.319535 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:32.323647 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:32.323656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:32.323663 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:32.323709 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:32.329105 (+ 5396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:32.330190 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:32.330195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:32.330198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:32.330357 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:32.330966 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:32.330971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:32.330975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:32.331421 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:32.331687 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:32.333003 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0927 22:58:32.333264 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:58:32.334332 30787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:32.336854 30787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:32.346681 30787 catalog_manager.cc:1347] Generated new cluster ID: 7ebc02dd70494115a86ea278a113aa04
I20260927 22:58:32.346807 30787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:32.370731 30787 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:32.372390 30787 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:32.380460 30787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6023e2f1b5b41199eded64a7412e947: Generated new TSK 0
I20260927 22:58:32.381345 30787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:32.399621 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30695
I20260927 22:58:32.401612 30757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:32.497897 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30695
W20260927 22:58:32.514985 30680 connection.cc:535] client connection to 127.8.204.190:37493 recv error: Network error: failed to read from TLS socket (remote: 127.8.204.190:37493): Connection reset by peer (error 104)
2026-09-27T22:58:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:32 dist-test-slave-40vx krb5kdc[30794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:32 dist-test-slave-40vx krb5kdc[30794](info): set up 2 sockets
Sep 27 22:58:32 dist-test-slave-40vx krb5kdc[30794](info): commencing operation
krb5kdc: starting...
W20260927 22:58:33.603860 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:33 dist-test-slave-40vx krb5kdc[30794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:33.850314 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44159
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:46223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:44159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:34.176875 30807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:34.177175 30807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:34.177338 30807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:34.184581 30807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:34.184698 30807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:34.184784 30807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:34.184870 30807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:34.184937 30807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:34.195075 30807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:46223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:44159
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:44159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:34.196422 30807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:34.197809 30807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:34.210796 30813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:34.210820 30812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:34.211170 30815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:34.211753 30807 server_base.cc:1034] running on GCE node
I20260927 22:58:34.212618 30807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:34.214069 30807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:34.215458 30807 hybrid_clock.cc:648] HybridClock initialized: now 1790549914215405 us; error 35 us; skew 500 ppm
I20260927 22:58:34.215828 30807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:34 dist-test-slave-40vx krb5kdc[30794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549914, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:34.218972 30807 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:34.220513 30807 webserver.cc:460] Webserver started at http://127.8.204.190:42987/ using document root <none> and password file <none>
I20260927 22:58:34.221211 30807 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:34.221357 30807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:34.221658 30807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:34.225109 30807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "e476066579fa4a388250bd7baa5c33da"
format_stamp: "Formatted at 2026-09-27 22:58:34 on dist-test-slave-40vx"
server_key: "8d321d595818d780b3e6106d085b1990"
server_key_iv: "ceb2abf60cf85afe0cda3932d8813fd8"
server_key_version: "encryptionkey@0"
I20260927 22:58:34.225893 30807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "e476066579fa4a388250bd7baa5c33da"
format_stamp: "Formatted at 2026-09-27 22:58:34 on dist-test-slave-40vx"
server_key: "8d321d595818d780b3e6106d085b1990"
server_key_iv: "ceb2abf60cf85afe0cda3932d8813fd8"
server_key_version: "encryptionkey@0"
I20260927 22:58:34.231076 30807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:58:34.234534 30822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:34.235931 30807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:34.236114 30807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "e476066579fa4a388250bd7baa5c33da"
format_stamp: "Formatted at 2026-09-27 22:58:34 on dist-test-slave-40vx"
server_key: "8d321d595818d780b3e6106d085b1990"
server_key_iv: "ceb2abf60cf85afe0cda3932d8813fd8"
server_key_version: "encryptionkey@0"
I20260927 22:58:34.236316 30807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:34.252911 30807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:34.256834 30807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:34.257107 30807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:34.275733 30807 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:44159
I20260927 22:58:34.275743 30873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:44159 every 8 connection(s)
I20260927 22:58:34.277038 30807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:34.280753 30874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:34.285401 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30807
I20260927 22:58:34.285586 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:34.286113 9010 external_mini_cluster.cc:1428] Setting key a71837737232fdaa99cc3a47227133ba
I20260927 22:58:34.292064 30874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da: Bootstrap starting.
I20260927 22:58:34.294987 30874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:34.296100 30874 log.cc:824] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:34.299517 30874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da: No bootstrap required, opened a new log
I20260927 22:58:34.300026 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:34.287890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44535 (local address 127.8.204.190:44159)
0927 22:58:34.288604 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:34.288615 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:34.288654 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:58:34.290738 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:34.290772 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:34.291229 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:34.291644 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:34.292156 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:34.293554 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:34.294822 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:34.295631 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:34.295826 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:34.297293 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:34.297301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:34.297314 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:34.298011 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:34.298116 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:34.298151 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:34.298317 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:34.298564 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 22:58:34.298732 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":323,"threads_started":1}
I20260927 22:58:34.303727 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:34.309381 30874 raft_consensus.cc:348] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476066579fa4a388250bd7baa5c33da" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44159 } }
I20260927 22:58:34.309835 30874 raft_consensus.cc:374] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:34.309978 30874 raft_consensus.cc:729] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e476066579fa4a388250bd7baa5c33da, State: Initialized, Role: FOLLOWER
I20260927 22:58:34.310673 30874 consensus_queue.cc:260] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476066579fa4a388250bd7baa5c33da" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44159 } }
I20260927 22:58:34.310963 30874 raft_consensus.cc:388] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:34.311106 30874 raft_consensus.cc:482] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:34.311304 30874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:34.314174 30874 raft_consensus.cc:504] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476066579fa4a388250bd7baa5c33da" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44159 } }
I20260927 22:58:34.314762 30874 leader_election.cc:302] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e476066579fa4a388250bd7baa5c33da; no voters:
I20260927 22:58:34.315518 30874 leader_election.cc:288] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:34.315644 30884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:34.316009 30877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:34.309074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57185 (local address 127.8.204.190:44159)
0927 22:58:34.309247 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:34.309252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:34.309603 (+ 351us) server_negotiation.cc:407] Connection header received
0927 22:58:34.309743 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:34.309750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:34.309832 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:34.309939 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:34.310483 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:34.311705 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:34.313071 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:34.313769 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:34.313911 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:34.314930 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:34.314936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:34.314941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:34.314961 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:34.315024 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:34.315057 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:34.315179 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:34.315396 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:58:34.315460 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:58:34.317966 30874 sys_catalog.cc:564] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:34.322710 30884 raft_consensus.cc:686] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [term 1 LEADER]: Becoming Leader. State: Replica: e476066579fa4a388250bd7baa5c33da, State: Running, Role: LEADER
I20260927 22:58:34.323134 30884 consensus_queue.cc:237] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476066579fa4a388250bd7baa5c33da" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44159 } }
I20260927 22:58:34.325978 30885 sys_catalog.cc:455] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e476066579fa4a388250bd7baa5c33da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476066579fa4a388250bd7baa5c33da" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44159 } } }
I20260927 22:58:34.326359 30885 sys_catalog.cc:458] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:34.326359 30886 sys_catalog.cc:455] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [sys.catalog]: SysCatalogTable state changed. Reason: New leader e476066579fa4a388250bd7baa5c33da. Latest consensus state: current_term: 1 leader_uuid: "e476066579fa4a388250bd7baa5c33da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e476066579fa4a388250bd7baa5c33da" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 44159 } } }
I20260927 22:58:34.328378 30886 sys_catalog.cc:458] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:34.330263 30898 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:58:34.331180 30899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:34.335456 30899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:34.343152 30899 catalog_manager.cc:1347] Generated new cluster ID: 0a27e15c25874777aa66ee6d3728cac8
I20260927 22:58:34.343299 30899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:34.356999 30899 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:34.358335 30899 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:34.367537 30899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e476066579fa4a388250bd7baa5c33da: Generated new TSK 0
I20260927 22:58:34.368371 30899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:34.385236 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30807
W20260927 22:58:34.386402 30869 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44535
W20260927 22:58:34.387198 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:34.387396 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30807
W20260927 22:58:34.403820 30792 connection.cc:535] client connection to 127.8.204.190:44159 recv error: Network error: failed to read from TLS socket (remote: 127.8.204.190:44159): Connection reset by peer (error 104)
2026-09-27T22:58:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:34 dist-test-slave-40vx krb5kdc[30905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:34 dist-test-slave-40vx krb5kdc[30905](info): set up 2 sockets
Sep 27 22:58:34 dist-test-slave-40vx krb5kdc[30905](info): commencing operation
krb5kdc: starting...
W20260927 22:58:35.483834 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:35 dist-test-slave-40vx krb5kdc[30905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:35.734126 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39715
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:39715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:36.064836 30918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:36.065197 30918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:36.065368 30918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:36.072508 30918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:36.072614 30918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:36.072661 30918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:36.072757 30918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:36.072827 30918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:36.082571 30918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:39715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:36.083912 30918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:36.085202 30918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:36.096819 30923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:36.096788 30924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:36.097311 30926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:36.097499 30918 server_base.cc:1034] running on GCE node
I20260927 22:58:36.098307 30918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:36.099735 30918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:36.100965 30918 hybrid_clock.cc:648] HybridClock initialized: now 1790549916100938 us; error 56 us; skew 500 ppm
I20260927 22:58:36.101370 30918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:36 dist-test-slave-40vx krb5kdc[30905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:36.104532 30918 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:36.105985 30918 webserver.cc:460] Webserver started at http://127.8.204.190:38415/ using document root <none> and password file <none>
I20260927 22:58:36.106647 30918 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:36.106791 30918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:36.107084 30918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:36.110049 30918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "cd3cca3950f348979e6254863931c450"
format_stamp: "Formatted at 2026-09-27 22:58:36 on dist-test-slave-40vx"
server_key: "d685e497ffc69ff3a719828299d1fbba"
server_key_iv: "2c6b6993a3a2e0f388ae7b0a989a11bd"
server_key_version: "encryptionkey@0"
I20260927 22:58:36.110824 30918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "cd3cca3950f348979e6254863931c450"
format_stamp: "Formatted at 2026-09-27 22:58:36 on dist-test-slave-40vx"
server_key: "d685e497ffc69ff3a719828299d1fbba"
server_key_iv: "2c6b6993a3a2e0f388ae7b0a989a11bd"
server_key_version: "encryptionkey@0"
I20260927 22:58:36.115922 30918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 22:58:36.119486 30933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:36.120801 30918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 22:58:36.120980 30918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "cd3cca3950f348979e6254863931c450"
format_stamp: "Formatted at 2026-09-27 22:58:36 on dist-test-slave-40vx"
server_key: "d685e497ffc69ff3a719828299d1fbba"
server_key_iv: "2c6b6993a3a2e0f388ae7b0a989a11bd"
server_key_version: "encryptionkey@0"
I20260927 22:58:36.121179 30918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:36.126547 30918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:36.130620 30918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:36.130906 30918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:36.149102 30918 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:39715
I20260927 22:58:36.149133 30984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:39715 every 8 connection(s)
I20260927 22:58:36.150497 30918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:36.154004 30985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:36.157433 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 30918
I20260927 22:58:36.157604 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:36.158103 9010 external_mini_cluster.cc:1428] Setting key fcafcebdd5ecb5d98d33a8a8b3fbd190
I20260927 22:58:36.165688 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:36.159866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50585 (local address 127.8.204.190:39715)
0927 22:58:36.160631 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:36.160642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:36.160682 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:58:36.162289 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:36.162310 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:36.162657 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:36.162972 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:36.164159 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:36.164165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:36.164177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:36.164191 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:36.164252 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:36.164275 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:36.164445 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:36.164593 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:58:36.164813 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":368,"threads_started":1}
I20260927 22:58:36.166615 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450: Bootstrap starting.
I20260927 22:58:36.169246 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:36.170246 30985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:36.171581 30985 log.cc:824] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:36.175071 30985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450: No bootstrap required, opened a new log
I20260927 22:58:36.177675 30988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:36.174760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44505 (local address 127.8.204.190:39715)
0927 22:58:36.174964 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:36.174971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:36.175378 (+ 407us) server_negotiation.cc:407] Connection header received
0927 22:58:36.175623 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:36.175627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:36.175691 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:36.175774 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:36.176823 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:36.176827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:36.176830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:36.176843 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:36.176876 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:36.176890 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:36.176975 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:36.177203 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:58:36.177254 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 22:58:36.184506 30985 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3cca3950f348979e6254863931c450" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39715 } }
I20260927 22:58:36.184898 30985 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:36.185026 30985 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3cca3950f348979e6254863931c450, State: Initialized, Role: FOLLOWER
I20260927 22:58:36.185668 30985 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3cca3950f348979e6254863931c450" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39715 } }
I20260927 22:58:36.185890 30985 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:36.186061 30985 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:36.186246 30985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:36.189678 30985 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3cca3950f348979e6254863931c450" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39715 } }
I20260927 22:58:36.190253 30985 leader_election.cc:302] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd3cca3950f348979e6254863931c450; no voters:
I20260927 22:58:36.190891 30985 leader_election.cc:288] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:36.190965 30995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:36.192250 30995 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [term 1 LEADER]: Becoming Leader. State: Replica: cd3cca3950f348979e6254863931c450, State: Running, Role: LEADER
I20260927 22:58:36.192746 30995 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3cca3950f348979e6254863931c450" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39715 } }
I20260927 22:58:36.193317 30985 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:36.195858 30996 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd3cca3950f348979e6254863931c450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3cca3950f348979e6254863931c450" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39715 } } }
I20260927 22:58:36.196453 30996 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:36.198699 31003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:36.198771 30997 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd3cca3950f348979e6254863931c450. Latest consensus state: current_term: 1 leader_uuid: "cd3cca3950f348979e6254863931c450" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd3cca3950f348979e6254863931c450" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39715 } } }
I20260927 22:58:36.199143 30997 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:36.204123 31003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:36.212740 31003 catalog_manager.cc:1347] Generated new cluster ID: 9fd5618046644272aaa02d6613e1b72d
I20260927 22:58:36.212850 31003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:36.252298 31003 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:36.252776 31003 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:36.264670 31003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd3cca3950f348979e6254863931c450: Generated new TSK 0
I20260927 22:58:36.265667 31003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:36.277033 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30918
W20260927 22:58:36.278023 30980 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50585
W20260927 22:58:36.278702 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 30918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:36.278813 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 30918
2026-09-27T22:58:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:36 dist-test-slave-40vx krb5kdc[31016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:36 dist-test-slave-40vx krb5kdc[31016](info): set up 2 sockets
Sep 27 22:58:36 dist-test-slave-40vx krb5kdc[31016](info): commencing operation
krb5kdc: starting...
W20260927 22:58:37.375821 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:37 dist-test-slave-40vx krb5kdc[31016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:37.622713 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37233
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:37.952667 31029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:37.952970 31029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:37.953063 31029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:37.960433 31029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:37.960558 31029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:37.960605 31029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:37.960666 31029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:37.960703 31029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:37.971076 31029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:37.972462 31029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:37.973723 31029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:37.985534 31037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:37.985598 31034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:37.985535 31035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:37.986299 31029 server_base.cc:1034] running on GCE node
I20260927 22:58:37.987125 31029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:37.988552 31029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:37.989786 31029 hybrid_clock.cc:648] HybridClock initialized: now 1790549917989750 us; error 49 us; skew 500 ppm
I20260927 22:58:37.990165 31029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:37 dist-test-slave-40vx krb5kdc[31016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549917, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:38.000731 31029 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:38.002286 31029 webserver.cc:460] Webserver started at http://127.8.204.190:33731/ using document root <none> and password file <none>
I20260927 22:58:38.002987 31029 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:38.003153 31029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:38.003520 31029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:38.006614 31029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "d9ce5e3b050b4e24ade65911ba9fff7b"
format_stamp: "Formatted at 2026-09-27 22:58:38 on dist-test-slave-40vx"
server_key: "bc0089ad6970ba1d4ae95f46d3b9d5c6"
server_key_iv: "f9abf8a55ccf4fde8911b6bb3743b458"
server_key_version: "encryptionkey@0"
I20260927 22:58:38.007488 31029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "d9ce5e3b050b4e24ade65911ba9fff7b"
format_stamp: "Formatted at 2026-09-27 22:58:38 on dist-test-slave-40vx"
server_key: "bc0089ad6970ba1d4ae95f46d3b9d5c6"
server_key_iv: "f9abf8a55ccf4fde8911b6bb3743b458"
server_key_version: "encryptionkey@0"
I20260927 22:58:38.012714 31029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:58:38.016307 31044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:38.017625 31029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:38.017820 31029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "d9ce5e3b050b4e24ade65911ba9fff7b"
format_stamp: "Formatted at 2026-09-27 22:58:38 on dist-test-slave-40vx"
server_key: "bc0089ad6970ba1d4ae95f46d3b9d5c6"
server_key_iv: "f9abf8a55ccf4fde8911b6bb3743b458"
server_key_version: "encryptionkey@0"
I20260927 22:58:38.018023 31029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:38.039440 31029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:38.043983 31029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:38.044277 31029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:38.063042 31029 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:37233
I20260927 22:58:38.063064 31095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:37233 every 8 connection(s)
I20260927 22:58:38.064514 31029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:38.066385 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31029
I20260927 22:58:38.066565 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:38.067013 9010 external_mini_cluster.cc:1428] Setting key 962aa387435a903760c3756cf993ffec
I20260927 22:58:38.068729 31096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:58:38 dist-test-slave-40vx krb5kdc[31016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:38.082109 31096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b: Bootstrap starting.
I20260927 22:58:38.085397 31096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:38.086486 31096 log.cc:824] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:38.089661 31096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b: No bootstrap required, opened a new log
I20260927 22:58:38.090430 31099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:38.068917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35843 (local address 127.8.204.190:37233)
0927 22:58:38.069712 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:38.069727 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:38.069774 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:58:38.071894 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:38.071922 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:38.072444 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:38.072889 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:38.074127 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:38.076163 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:38.077420 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:38.078435 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:38.078613 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:38.082991 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:38.082999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:38.083012 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:38.083045 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:38.086263 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:38.087046 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:38.087055 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:38.087066 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:38.087341 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:38.087722 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:38.087726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:38.087729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:38.088284 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:38.088490 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:38.088942 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 22:58:38.089310 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":337,"threads_started":1}
I20260927 22:58:38.093981 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:38.097230 31096 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37233 } }
I20260927 22:58:38.097558 31096 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:38.097672 31096 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ce5e3b050b4e24ade65911ba9fff7b, State: Initialized, Role: FOLLOWER
I20260927 22:58:38.098335 31096 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37233 } }
I20260927 22:58:38.098677 31096 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:38.098829 31096 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:38.099030 31096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:38.103214 31096 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37233 } }
I20260927 22:58:38.103838 31096 leader_election.cc:302] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ce5e3b050b4e24ade65911ba9fff7b; no voters:
I20260927 22:58:38.104454 31096 leader_election.cc:288] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:38.104586 31106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:38.105295 31106 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [term 1 LEADER]: Becoming Leader. State: Replica: d9ce5e3b050b4e24ade65911ba9fff7b, State: Running, Role: LEADER
I20260927 22:58:38.105935 31106 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37233 } }
I20260927 22:58:38.107970 31096 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:38.108578 31107 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ce5e3b050b4e24ade65911ba9fff7b. Latest consensus state: current_term: 1 leader_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37233 } } }
I20260927 22:58:38.108896 31107 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:38.109382 31108 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ce5e3b050b4e24ade65911ba9fff7b" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37233 } } }
I20260927 22:58:38.109699 31108 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:38.109900 31111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:38.120141 31099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:38.098115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42703 (local address 127.8.204.190:37233)
0927 22:58:38.098288 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:38.098293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:38.103836 (+ 5543us) server_negotiation.cc:407] Connection header received
0927 22:58:38.104054 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:38.104059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:38.104135 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:38.104259 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:38.106236 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:38.107426 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:38.109205 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:38.109980 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:38.110100 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:38.113502 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:38.113512 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:38.113520 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:38.113568 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:38.117507 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:38.118110 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:38.118115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:38.118119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:38.118249 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:38.118681 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:38.118686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:38.118691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:38.118988 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:38.119315 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:38.119680 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 22:58:38.119868 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:58:38.125996 31111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:38.134553 31111 catalog_manager.cc:1347] Generated new cluster ID: f51967caf81c4c75a33a3bfc1fd7a88e
I20260927 22:58:38.134732 31111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:38.139889 31124 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:58:38.158939 31111 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:38.160321 31111 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:38.168264 31111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ce5e3b050b4e24ade65911ba9fff7b: Generated new TSK 0
I20260927 22:58:38.169116 31111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:38.177804 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31029
I20260927 22:58:38.179603 31091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:38.269853 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31029
2026-09-27T22:58:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:38 dist-test-slave-40vx krb5kdc[31128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:38 dist-test-slave-40vx krb5kdc[31128](info): set up 2 sockets
Sep 27 22:58:38 dist-test-slave-40vx krb5kdc[31128](info): commencing operation
krb5kdc: starting...
W20260927 22:58:39.367820 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:39 dist-test-slave-40vx krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:39.622735 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:43127
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:43851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:43127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:39.949868 31141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:39.950157 31141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:39.950315 31141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:39.957484 31141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:39.957587 31141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:39.957687 31141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:39.957791 31141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:39.957877 31141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:39.967597 31141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:43851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:43127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:43127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:39.968870 31141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:39.970170 31141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:39.982543 31147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:39.982652 31146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:39.982656 31149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:39.983312 31141 server_base.cc:1034] running on GCE node
I20260927 22:58:39.984143 31141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:39.985533 31141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:39.986753 31141 hybrid_clock.cc:648] HybridClock initialized: now 1790549919986695 us; error 58 us; skew 500 ppm
I20260927 22:58:39.987097 31141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:39 dist-test-slave-40vx krb5kdc[31128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549919, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:39.990226 31141 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:39.991770 31141 webserver.cc:460] Webserver started at http://127.8.204.190:45369/ using document root <none> and password file <none>
I20260927 22:58:39.992429 31141 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:39.992571 31141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:39.992868 31141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:39.995724 31141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "f01090eda0234df1a32fa04ec1d322ab"
format_stamp: "Formatted at 2026-09-27 22:58:39 on dist-test-slave-40vx"
server_key: "e68e6b7d2c62562afbc21e3cb1878fcc"
server_key_iv: "1762a95539f013e428b65e342fa61be7"
server_key_version: "encryptionkey@0"
I20260927 22:58:39.996451 31141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "f01090eda0234df1a32fa04ec1d322ab"
format_stamp: "Formatted at 2026-09-27 22:58:39 on dist-test-slave-40vx"
server_key: "e68e6b7d2c62562afbc21e3cb1878fcc"
server_key_iv: "1762a95539f013e428b65e342fa61be7"
server_key_version: "encryptionkey@0"
I20260927 22:58:40.001575 31141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:58:40.005705 31156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:40.007068 31141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:58:40.007254 31141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "f01090eda0234df1a32fa04ec1d322ab"
format_stamp: "Formatted at 2026-09-27 22:58:39 on dist-test-slave-40vx"
server_key: "e68e6b7d2c62562afbc21e3cb1878fcc"
server_key_iv: "1762a95539f013e428b65e342fa61be7"
server_key_version: "encryptionkey@0"
I20260927 22:58:40.007508 31141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:40.026117 31141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:40.030668 31141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:40.030901 31141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:40.048441 31141 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:43127
I20260927 22:58:40.048462 31207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:43127 every 8 connection(s)
I20260927 22:58:40.049817 31141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:40.051848 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31141
I20260927 22:58:40.052027 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:40.052596 9010 external_mini_cluster.cc:1428] Setting key cca4415706487c00d1e834169bada5e6
I20260927 22:58:40.054432 31208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:40.065418 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:40.054317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54769 (local address 127.8.204.190:43127)
0927 22:58:40.055139 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:40.055150 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:40.055190 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:58:40.056799 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:40.056819 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:40.057206 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:40.057545 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:40.058069 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:40.059423 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:40.061202 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:40.061847 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:40.061966 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:40.063130 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:40.063137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:40.063149 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:40.063761 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:40.063837 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:40.063864 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:40.063950 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:40.064196 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 22:58:40.064471 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":394,"threads_started":1}
I20260927 22:58:40.066766 31208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab: Bootstrap starting.
I20260927 22:58:40.068931 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:40.070001 31208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:40.070958 31208 log.cc:824] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:40.073875 31208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab: No bootstrap required, opened a new log
I20260927 22:58:40.079824 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:40.073270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57359 (local address 127.8.204.190:43127)
0927 22:58:40.073473 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:40.073478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:40.073957 (+ 479us) server_negotiation.cc:407] Connection header received
0927 22:58:40.074138 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:40.074146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:40.074216 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:40.074353 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:40.074787 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:40.075899 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:40.077152 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:40.077873 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:40.077994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:40.079018 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:40.079025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:40.079030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:40.079052 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:40.079101 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:40.079121 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:40.079231 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:40.079392 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 22:58:40.079488 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:58:40.082666 31208 raft_consensus.cc:348] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01090eda0234df1a32fa04ec1d322ab" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 43127 } }
I20260927 22:58:40.083041 31208 raft_consensus.cc:374] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:40.083161 31208 raft_consensus.cc:729] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01090eda0234df1a32fa04ec1d322ab, State: Initialized, Role: FOLLOWER
I20260927 22:58:40.083766 31208 consensus_queue.cc:260] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01090eda0234df1a32fa04ec1d322ab" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 43127 } }
I20260927 22:58:40.083998 31208 raft_consensus.cc:388] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:40.084151 31208 raft_consensus.cc:482] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:40.084331 31208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:40.086460 31208 raft_consensus.cc:504] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01090eda0234df1a32fa04ec1d322ab" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 43127 } }
I20260927 22:58:40.086974 31208 leader_election.cc:302] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f01090eda0234df1a32fa04ec1d322ab; no voters:
I20260927 22:58:40.087599 31208 leader_election.cc:288] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:40.087726 31218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:40.088410 31218 raft_consensus.cc:686] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [term 1 LEADER]: Becoming Leader. State: Replica: f01090eda0234df1a32fa04ec1d322ab, State: Running, Role: LEADER
I20260927 22:58:40.088979 31218 consensus_queue.cc:237] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01090eda0234df1a32fa04ec1d322ab" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 43127 } }
I20260927 22:58:40.089674 31208 sys_catalog.cc:564] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:40.092211 31220 sys_catalog.cc:455] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f01090eda0234df1a32fa04ec1d322ab. Latest consensus state: current_term: 1 leader_uuid: "f01090eda0234df1a32fa04ec1d322ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01090eda0234df1a32fa04ec1d322ab" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 43127 } } }
I20260927 22:58:40.092203 31219 sys_catalog.cc:455] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f01090eda0234df1a32fa04ec1d322ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f01090eda0234df1a32fa04ec1d322ab" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 43127 } } }
I20260927 22:58:40.092617 31220 sys_catalog.cc:458] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:40.092656 31219 sys_catalog.cc:458] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:40.094318 31226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:40.101070 31226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:40.109853 31226 catalog_manager.cc:1347] Generated new cluster ID: 6629d88b40b94a2abfc56fe4c5c031ab
I20260927 22:58:40.109975 31226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:40.123952 31226 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:40.125762 31226 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:40.137826 31226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f01090eda0234df1a32fa04ec1d322ab: Generated new TSK 0
I20260927 22:58:40.138952 31226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:40.145080 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31141
W20260927 22:58:40.146330 31203 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54769
W20260927 22:58:40.147019 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31141: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:40.147162 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31141
2026-09-27T22:58:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:40 dist-test-slave-40vx krb5kdc[31239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:40 dist-test-slave-40vx krb5kdc[31239](info): set up 2 sockets
Sep 27 22:58:40 dist-test-slave-40vx krb5kdc[31239](info): commencing operation
krb5kdc: starting...
W20260927 22:58:41.251829 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:41 dist-test-slave-40vx krb5kdc[31239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:41.496127 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:46697
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:40665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:41.824404 31252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:41.824756 31252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:41.824916 31252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:41.832405 31252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:41.832566 31252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:41.832667 31252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:41.832773 31252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:41.832855 31252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:41.842926 31252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:40665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:41.844380 31252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:41.845764 31252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:41.857224 31258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:41.857357 31257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:41.857512 31260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:41.858319 31252 server_base.cc:1034] running on GCE node
I20260927 22:58:41.859109 31252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:41.860549 31252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:41.861791 31252 hybrid_clock.cc:648] HybridClock initialized: now 1790549921861738 us; error 46 us; skew 500 ppm
I20260927 22:58:41.862156 31252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:41 dist-test-slave-40vx krb5kdc[31239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549921, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:41.865401 31252 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:41.866914 31252 webserver.cc:460] Webserver started at http://127.8.204.190:42941/ using document root <none> and password file <none>
I20260927 22:58:41.867686 31252 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:41.867835 31252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:41.868160 31252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:41.871088 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "43e5566f814542499938b3088ffa9170"
format_stamp: "Formatted at 2026-09-27 22:58:41 on dist-test-slave-40vx"
server_key: "a09cba2692fb7a7964505c3b033d3e12"
server_key_iv: "9253f938f965b2ef5dcf17dd362e1ddb"
server_key_version: "encryptionkey@0"
I20260927 22:58:41.871924 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "43e5566f814542499938b3088ffa9170"
format_stamp: "Formatted at 2026-09-27 22:58:41 on dist-test-slave-40vx"
server_key: "a09cba2692fb7a7964505c3b033d3e12"
server_key_iv: "9253f938f965b2ef5dcf17dd362e1ddb"
server_key_version: "encryptionkey@0"
I20260927 22:58:41.876989 31252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:58:41.880492 31267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:41.881791 31252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:41.881989 31252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "43e5566f814542499938b3088ffa9170"
format_stamp: "Formatted at 2026-09-27 22:58:41 on dist-test-slave-40vx"
server_key: "a09cba2692fb7a7964505c3b033d3e12"
server_key_iv: "9253f938f965b2ef5dcf17dd362e1ddb"
server_key_version: "encryptionkey@0"
I20260927 22:58:41.882200 31252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:41.888094 31252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:41.892407 31252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:41.892706 31252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:41.911093 31252 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:46697
I20260927 22:58:41.911131 31318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:46697 every 8 connection(s)
I20260927 22:58:41.912578 31252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:41.916383 31319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:41.919509 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31252
I20260927 22:58:41.919692 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:41.920145 9010 external_mini_cluster.cc:1428] Setting key 8ab6900cb8d150534e7a761129171438
I20260927 22:58:41.928189 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:41.922132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50377 (local address 127.8.204.190:46697)
0927 22:58:41.922960 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:41.922976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:41.923017 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:58:41.924636 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:41.924662 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:41.925036 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:41.925376 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:41.926479 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:41.926485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:41.926498 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:41.926511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:41.926565 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:41.926587 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:41.926754 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:41.926899 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 22:58:41.927099 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":378,"threads_started":1}
I20260927 22:58:41.931803 31319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170: Bootstrap starting.
I20260927 22:58:41.931950 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:41.935812 31319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:41.937211 31319 log.cc:824] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:41.940028 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:41.937215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50057 (local address 127.8.204.190:46697)
0927 22:58:41.937453 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:41.937463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:41.937987 (+ 524us) server_negotiation.cc:407] Connection header received
0927 22:58:41.938140 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:41.938147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:41.938226 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:41.938360 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:41.939359 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:41.939363 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:41.939366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:58:41.939382 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:41.939418 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:58:41.939433 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:41.939521 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:41.939667 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:58:41.939723 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:58:41.940861 31319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170: No bootstrap required, opened a new log
I20260927 22:58:41.948522 31319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e5566f814542499938b3088ffa9170" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46697 } }
I20260927 22:58:41.948861 31319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:58:41.948931 31319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43e5566f814542499938b3088ffa9170, State: Initialized, Role: FOLLOWER
I20260927 22:58:41.949426 31319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e5566f814542499938b3088ffa9170" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46697 } }
I20260927 22:58:41.949620 31319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:58:41.949771 31319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:58:41.949899 31319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:41.952526 31319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e5566f814542499938b3088ffa9170" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46697 } }
I20260927 22:58:41.952950 31319 leader_election.cc:302] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43e5566f814542499938b3088ffa9170; no voters:
I20260927 22:58:41.953438 31319 leader_election.cc:288] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:58:41.953569 31329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:41.954257 31329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [term 1 LEADER]: Becoming Leader. State: Replica: 43e5566f814542499938b3088ffa9170, State: Running, Role: LEADER
I20260927 22:58:41.954743 31329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e5566f814542499938b3088ffa9170" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46697 } }
I20260927 22:58:41.955567 31319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:41.957597 31330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43e5566f814542499938b3088ffa9170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e5566f814542499938b3088ffa9170" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46697 } } }
I20260927 22:58:41.957628 31331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43e5566f814542499938b3088ffa9170. Latest consensus state: current_term: 1 leader_uuid: "43e5566f814542499938b3088ffa9170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43e5566f814542499938b3088ffa9170" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 46697 } } }
I20260927 22:58:41.958127 31330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:41.958127 31331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:41.958910 31338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:58:41.966014 31338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:41.974565 31338 catalog_manager.cc:1347] Generated new cluster ID: 912aa7256b2141b7bd156f4134ebda21
I20260927 22:58:41.974690 31338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:41.992556 31338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:41.993037 31338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:42.003332 31338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43e5566f814542499938b3088ffa9170: Generated new TSK 0
I20260927 22:58:42.004479 31338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:42.020875 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31252
W20260927 22:58:42.021922 31314 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50377
W20260927 22:58:42.022661 9010 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:58:42.022815 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31252
2026-09-27T22:58:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1872 ms)
[----------] 18 tests from AuthTokenIssuingTest (34687 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:42 dist-test-slave-40vx krb5kdc[31350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:42 dist-test-slave-40vx krb5kdc[31350](info): set up 2 sockets
Sep 27 22:58:42 dist-test-slave-40vx krb5kdc[31350](info): commencing operation
krb5kdc: starting...
W20260927 22:58:43.120467 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:43 dist-test-slave-40vx krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:43.374846 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37649
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37649,127.8.204.189:42017,127.8.204.188:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:43.698525 31363 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:58:43.698879 31363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:43.699004 31363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:43.699141 31363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:43.706308 31363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:43.706400 31363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:43.706530 31363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:43.706606 31363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:43.706696 31363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:43.716728 31363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37649,127.8.204.189:42017,127.8.204.188:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:37649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:43.718061 31363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:43.719426 31363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:43.731608 31371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:43.731714 31369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:43.731758 31368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:43.732496 31363 server_base.cc:1034] running on GCE node
I20260927 22:58:43.733178 31363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:43.734457 31363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:43.735692 31363 hybrid_clock.cc:648] HybridClock initialized: now 1790549923735588 us; error 49 us; skew 500 ppm
I20260927 22:58:43.736053 31363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:43 dist-test-slave-40vx krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:43.745667 31363 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:43.747243 31363 webserver.cc:460] Webserver started at http://127.8.204.190:42241/ using document root <none> and password file <none>
I20260927 22:58:43.748042 31363 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:43.748206 31363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:43.748536 31363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:43.751569 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "286ac7bd56934e43b86473315160c0e1"
format_stamp: "Formatted at 2026-09-27 22:58:43 on dist-test-slave-40vx"
server_key: "d92accb06655e285f3b4e76e50794528"
server_key_iv: "f03feb262343e47092f5d30d2c226de2"
server_key_version: "encryptionkey@0"
I20260927 22:58:43.752365 31363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "286ac7bd56934e43b86473315160c0e1"
format_stamp: "Formatted at 2026-09-27 22:58:43 on dist-test-slave-40vx"
server_key: "d92accb06655e285f3b4e76e50794528"
server_key_iv: "f03feb262343e47092f5d30d2c226de2"
server_key_version: "encryptionkey@0"
I20260927 22:58:43.757952 31363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:58:43.761477 31378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:43.762961 31363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:58:43.763166 31363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "286ac7bd56934e43b86473315160c0e1"
format_stamp: "Formatted at 2026-09-27 22:58:43 on dist-test-slave-40vx"
server_key: "d92accb06655e285f3b4e76e50794528"
server_key_iv: "f03feb262343e47092f5d30d2c226de2"
server_key_version: "encryptionkey@0"
I20260927 22:58:43.763464 31363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:43.777869 31363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:43.782047 31363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:43.782326 31363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:43.800544 31363 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:37649
I20260927 22:58:43.800565 31429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:37649 every 8 connection(s)
I20260927 22:58:43.801884 31363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:43.806051 31430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:43.808809 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31363
I20260927 22:58:43.809032 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:43.809824 9010 external_mini_cluster.cc:1428] Setting key f300e69a4c7fc8afd99ecd447a536f02
I20260927 22:58:43.816282 31430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:43 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.189@KRBTEST.COM" created.
I20260927 22:58:43.844007 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.820363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37649 (local address 127.0.0.1:52170)
0927 22:58:43.821327 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.821378 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:43.824792 (+ 3414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:43.825835 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:43.825844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:43.826905 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:43.827414 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:43.827441 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:43.829123 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:43.829131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:43.831074 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:43.831093 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:43.832168 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:43.832178 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:43.832404 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:43.833128 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:43.833164 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:43.835975 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:43.839596 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:43.839605 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:43.839615 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:43.840017 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:43.840592 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:43.840595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:43.840597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:43.840737 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:43.841730 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:43.841736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:43.841943 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 22:58:43.842160 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 22:58:43.842852 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":210,"thread_start_us":390,"threads_started":1}
I20260927 22:58:43.844058 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.820363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.8.204.190:37649)
0927 22:58:43.823718 (+ 3355us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:43.823728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:43.823782 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:58:43.825217 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:43.825223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:43.825485 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:43.825750 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:43.827668 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:43.828947 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:43.831316 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:43.831902 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:43.832047 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:43.836222 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:43.836230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:43.836244 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:43.836272 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:43.839456 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:43.840143 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:43.840150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:43.840152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:43.840482 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:43.840871 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:43.840875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:43.840877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:43.841587 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:43.841795 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:43.842364 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 22:58:43.842732 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3133,"spinlock_wait_cycles":17280,"thread_start_us":2911,"threads_started":1}
I20260927 22:58:43.847213 31430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:43.848563 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.848123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33242)
0927 22:58:43.848329 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.848438 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 22:58:43.848871 31381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
I20260927 22:58:43.850459 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.850105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33244)
0927 22:58:43.850257 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.850331 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 22:58:43.850678 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42017: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab.
Entry for principal kudu/127.8.204.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab.
I20260927 22:58:43.874212 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:42017
--webserver_interface=127.8.204.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab
--principal=kudu/127.8.204.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37649,127.8.204.189:42017,127.8.204.188:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:58:43.900012 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } attempt: 1
I20260927 22:58:43.901329 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.900757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33248)
0927 22:58:43.900959 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.901092 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:58:43.902323 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.901965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33254)
0927 22:58:43.902110 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.902184 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:58:43.902503 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42017: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
I20260927 22:58:43.970757 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } attempt: 2
I20260927 22:58:43.971975 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.971470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33256)
0927 22:58:43.971734 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.971827 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:58:43.973121 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:43.972736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33262)
0927 22:58:43.972918 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:43.972989 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:58:43.973338 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42017: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
I20260927 22:58:44.064574 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } attempt: 3
I20260927 22:58:44.065711 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.065271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33276)
0927 22:58:44.065477 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.065580 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:58:44.066902 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.066516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33284)
0927 22:58:44.066676 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.066764 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 22:58:44.067129 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42017: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
W20260927 22:58:44.200343 31437 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:58:44.200691 31437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:44.200832 31437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:44.200989 31437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:44.208119 31437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:44.208315 31437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:44.208449 31437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:44.208531 31437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:44.208614 31437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:44.210426 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } attempt: 4
I20260927 22:58:44.211575 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.211022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33296)
0927 22:58:44.211347 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.211442 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260927 22:58:44.212589 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.212254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33312)
0927 22:58:44.212392 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.212478 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:58:44.212781 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:42017: Network error: Client connection negotiation failed: client connection to 127.8.204.189:42017: connect: Connection refused (error 111)
I20260927 22:58:44.218779 31437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37649,127.8.204.189:42017,127.8.204.188:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab
--principal=kudu/127.8.204.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:44.220129 31437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:44.221443 31437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:44.233265 31445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:44.233290 31443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:44.233569 31442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:44.233978 31437 server_base.cc:1034] running on GCE node
I20260927 22:58:44.234809 31437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:44.236219 31437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:44.237432 31437 hybrid_clock.cc:648] HybridClock initialized: now 1790549924237383 us; error 34 us; skew 500 ppm
I20260927 22:58:44.237753 31437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:44.240967 31437 init.cc:377] Logged in from keytab as kudu/127.8.204.189@KRBTEST.COM (short username kudu)
I20260927 22:58:44.242441 31437 webserver.cc:460] Webserver started at http://127.8.204.189:45009/ using document root <none> and password file <none>
I20260927 22:58:44.243108 31437 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:44.243247 31437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:44.243705 31437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:44.246457 31437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data/instance:
uuid: "903cb17c2b374959b32dc11121e90769"
format_stamp: "Formatted at 2026-09-27 22:58:44 on dist-test-slave-40vx"
server_key: "8329135d5fe7cb1f0974878a4f5564cb"
server_key_iv: "849e9c9b67280d5e4bb817b88850b902"
server_key_version: "encryptionkey@0"
I20260927 22:58:44.247244 31437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal/instance:
uuid: "903cb17c2b374959b32dc11121e90769"
format_stamp: "Formatted at 2026-09-27 22:58:44 on dist-test-slave-40vx"
server_key: "8329135d5fe7cb1f0974878a4f5564cb"
server_key_iv: "849e9c9b67280d5e4bb817b88850b902"
server_key_version: "encryptionkey@0"
I20260927 22:58:44.252198 31437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:58:44.255517 31452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:44.256821 31437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 22:58:44.256999 31437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal
uuid: "903cb17c2b374959b32dc11121e90769"
format_stamp: "Formatted at 2026-09-27 22:58:44 on dist-test-slave-40vx"
server_key: "8329135d5fe7cb1f0974878a4f5564cb"
server_key_iv: "849e9c9b67280d5e4bb817b88850b902"
server_key_version: "encryptionkey@0"
I20260927 22:58:44.257262 31437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:44.274516 31437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:44.278803 31437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:44.279069 31437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:44.298228 31437 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.189:42017
I20260927 22:58:44.298235 31503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.189:42017 every 8 connection(s)
I20260927 22:58:44.299666 31437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
I20260927 22:58:44.303318 31504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:44.308207 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31437
I20260927 22:58:44.308437 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-1/wal/instance
I20260927 22:58:44.308861 9010 external_mini_cluster.cc:1428] Setting key a903397775cde135235eada0657f4ee1
I20260927 22:58:44.309605 31504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:44.332691 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.312118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.8.204.190:37649)
0927 22:58:44.312314 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.312318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.313536 (+ 1218us) server_negotiation.cc:407] Connection header received
0927 22:58:44.315545 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.315551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.315641 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.315753 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.317642 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.318345 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.320828 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.321273 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.321375 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.325434 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.325439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.325442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.325475 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.329194 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.329913 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.329921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.329928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.330086 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.330508 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.330515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.330521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.330937 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.331104 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.331954 (+ 850us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.332233 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:58:44.333287 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.312648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37649 (local address 127.0.0.1:52186)
0927 22:58:44.313371 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.313409 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.315218 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.315981 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.315989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.316653 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.317454 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.317486 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.318508 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.318517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.320636 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.320658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.321506 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.321515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.321824 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.322546 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.322578 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.325168 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.329333 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.329339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.329351 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.329771 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.330198 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.330202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.330204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.330382 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.331084 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.331093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.331531 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.331767 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.332150 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":278,"threads_started":1}
I20260927 22:58:44.335090 31504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.204.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.188@KRBTEST.COM" created.
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:58:44.349677 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.335901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33324)
0927 22:58:44.336085 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.336100 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.336187 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.337201 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.337205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.337644 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.337771 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.337779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.339393 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.339401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.340240 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.340251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.341076 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.341081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.341301 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.341904 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.341920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.343947 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.347141 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.347147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.347151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.347591 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.347944 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.347951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.347954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.348094 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.348686 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.348695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.348838 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.348989 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.349271 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":5}
I20260927 22:58:44.349740 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.336161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.8.204.189:42017)
0927 22:58:44.336805 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.336812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.336838 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:58:44.336878 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.336884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.337060 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.337213 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.337954 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.339215 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.340378 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.340935 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.341068 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.344155 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.344165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.344180 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.344206 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.347014 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.347713 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.347722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.347726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.347842 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.348204 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.348212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.348215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.348566 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.348783 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.349120 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.349383 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":509,"thread_start_us":385,"threads_started":1}
I20260927 22:58:44.352600 31504 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:44.353652 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.353189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37552)
0927 22:58:44.353408 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.353505 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:58:44.353878 31455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.355424 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.355068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37558)
0927 22:58:44.355197 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.355318 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 22:58:44.355648 31504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab.
Entry for principal kudu/127.8.204.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab.
I20260927 22:58:44.370165 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:39121
--webserver_interface=127.8.204.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab
--principal=kudu/127.8.204.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:37649,127.8.204.189:42017,127.8.204.188:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:58:44.404955 31504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 1
I20260927 22:58:44.406188 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.405634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37574)
0927 22:58:44.405866 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.405996 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:58:44.407243 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.406867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37590)
0927 22:58:44.407031 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.407119 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 22:58:44.407500 31504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.475761 31504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 2
I20260927 22:58:44.477182 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.476522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37600)
0927 22:58:44.476796 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.476963 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 22:58:44.478319 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.477910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37606)
0927 22:58:44.478072 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.478157 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:58:44.478544 31504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.512071 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } attempt: 5
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:58:44.526221 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.512929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.8.204.189:42017)
0927 22:58:44.513105 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.513110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.513173 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:58:44.513383 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.513387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.513455 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.513532 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.514499 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.515340 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.517138 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.517583 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.517719 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.521017 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.521022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.521025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.521056 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.523682 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.524367 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.524374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.524382 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.524499 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.524863 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.524869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.524873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.525150 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.525306 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.525675 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.525900 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:58:44.526221 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.512847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33326)
0927 22:58:44.513068 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.513090 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.513233 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.513598 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.513604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.514117 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.514345 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.514356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.515471 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.515478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.516974 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.516993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.517732 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.517738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.517972 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.518592 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.518611 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.520834 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.523845 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.523850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.523854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.524247 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.524609 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.524614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.524618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.524760 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.525276 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.525280 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.525432 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.525571 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.525877 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:58:44.527518 31430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:44.528625 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.528248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37618)
0927 22:58:44.528412 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.528493 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:58:44.529572 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.529251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37620)
0927 22:58:44.529388 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.529471 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:58:44.529793 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.569799 31504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 3
I20260927 22:58:44.571175 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.570568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37622)
0927 22:58:44.570859 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.570989 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 22:58:44.572472 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.571981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37626)
0927 22:58:44.572266 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.572351 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260927 22:58:44.572798 31504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.581003 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 1
I20260927 22:58:44.582293 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.581666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37642)
0927 22:58:44.581929 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.582055 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:58:44.583459 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.583007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37648)
0927 22:58:44.583214 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.583319 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:58:44.583741 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.651988 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 2
I20260927 22:58:44.653399 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.652776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37660)
0927 22:58:44.653068 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.653194 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:58:44.654562 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.654131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37662)
0927 22:58:44.654318 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.654412 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:58:44.654860 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
W20260927 22:58:44.703234 31511 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:58:44.703658 31511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:44.703783 31511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:44.703922 31511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:44.711055 31511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:44.711189 31511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:44.711309 31511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:44.711404 31511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:44.711493 31511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:44.716025 31504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 4
I20260927 22:58:44.717250 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.716695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37674)
0927 22:58:44.716947 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.717058 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:58:44.718308 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.717926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37678)
0927 22:58:44.718080 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.718183 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:58:44.718534 31504 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.722399 31511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:37649,127.8.204.189:42017,127.8.204.188:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab
--principal=kudu/127.8.204.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:44.723800 31511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:44.725076 31511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:44.736512 31517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:44.736595 31516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:44.736553 31519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:44.737268 31511 server_base.cc:1034] running on GCE node
I20260927 22:58:44.737915 31511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:44.739365 31511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:44.740590 31511 hybrid_clock.cc:648] HybridClock initialized: now 1790549924740529 us; error 41 us; skew 500 ppm
I20260927 22:58:44.740922 31511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:44.744061 31511 init.cc:377] Logged in from keytab as kudu/127.8.204.188@KRBTEST.COM (short username kudu)
I20260927 22:58:44.745476 31511 webserver.cc:460] Webserver started at http://127.8.204.188:34057/ using document root <none> and password file <none>
I20260927 22:58:44.746091 31511 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:44.746176 31511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:44.746438 31511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:44.749396 31511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data/instance:
uuid: "b40feb4628664cae80ab7839085de515"
format_stamp: "Formatted at 2026-09-27 22:58:44 on dist-test-slave-40vx"
server_key: "795d17e6fa71490e8dda49f63a1cfe9a"
server_key_iv: "bea0605290903c946c0c54e2f1371a84"
server_key_version: "encryptionkey@0"
I20260927 22:58:44.750140 31511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal/instance:
uuid: "b40feb4628664cae80ab7839085de515"
format_stamp: "Formatted at 2026-09-27 22:58:44 on dist-test-slave-40vx"
server_key: "795d17e6fa71490e8dda49f63a1cfe9a"
server_key_iv: "bea0605290903c946c0c54e2f1371a84"
server_key_version: "encryptionkey@0"
I20260927 22:58:44.755617 31511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 22:58:44.759011 31526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:44.760491 31511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 22:58:44.760685 31511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal
uuid: "b40feb4628664cae80ab7839085de515"
format_stamp: "Formatted at 2026-09-27 22:58:44 on dist-test-slave-40vx"
server_key: "795d17e6fa71490e8dda49f63a1cfe9a"
server_key_iv: "bea0605290903c946c0c54e2f1371a84"
server_key_version: "encryptionkey@0"
I20260927 22:58:44.760962 31511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:44.761053 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 3
I20260927 22:58:44.762097 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.761681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37692)
0927 22:58:44.761860 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.761951 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:58:44.763060 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.762734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37696)
0927 22:58:44.762876 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.762959 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:58:44.763240 31430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:39121: Network error: Client connection negotiation failed: client connection to 127.8.204.188:39121: connect: Connection refused (error 111)
I20260927 22:58:44.773108 31511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:44.777508 31511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:44.777791 31511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:44.795645 31511 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.188:39121
I20260927 22:58:44.795665 31577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.188:39121 every 8 connection(s)
I20260927 22:58:44.797073 31511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
I20260927 22:58:44.801116 31578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:44.806962 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31511
I20260927 22:58:44.807160 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790549572925352-9010-0/minicluster-data/master-2/wal/instance
I20260927 22:58:44.807614 9010 external_mini_cluster.cc:1428] Setting key 53773dccd05b6324a7f063dc1036d4b0
I20260927 22:58:44.810006 31578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:58:44.829376 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.812980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.8.204.190:37649)
0927 22:58:44.813196 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.813204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.813901 (+ 697us) server_negotiation.cc:407] Connection header received
0927 22:58:44.814533 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.814537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.814596 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.814668 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.816150 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.816869 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.818501 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.819106 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.819197 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.822957 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.822964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.822969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.823014 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.826069 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.826980 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.826986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.826990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.827128 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.827585 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.827592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.827595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.827879 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.828092 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.828771 (+ 679us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.829009 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:58:44.830446 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.812989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:37649 (local address 127.0.0.1:52194)
0927 22:58:44.813734 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.813773 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.814281 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.814753 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.814760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.815543 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.815981 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.816000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.817023 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.817029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.818364 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.818375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.819367 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.819375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.819592 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.820394 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.820415 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.822706 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.826224 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.826235 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.826252 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.826860 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.827244 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.827249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.827253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.827470 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.828063 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.828071 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.828420 (+ 349us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.828658 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.829186 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":345,"threads_started":1}
I20260927 22:58:44.830463 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.809595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.8.204.188:39121)
0927 22:58:44.811397 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.811413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.811816 (+ 403us) server_negotiation.cc:407] Connection header received
0927 22:58:44.814281 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.814314 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.814591 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.814791 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.816215 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.817568 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.818931 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.819640 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.819779 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.823993 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.824001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.824011 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.824038 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.826717 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.827404 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.827419 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.827421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.827547 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.827915 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.827919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.827921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.828462 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.828783 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.829069 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.829333 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1533,"thread_start_us":1284,"threads_started":1}
I20260927 22:58:44.832207 31578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:44.846788 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.833095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:42017 (local address 127.0.0.1:33334)
0927 22:58:44.833242 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.833262 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.833384 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.833990 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.833996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.834485 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.834706 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.834718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.835823 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.835827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.836917 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.836932 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.837622 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.837628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.837858 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.838467 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.838483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.840841 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.844392 (+ 3551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.844397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.844402 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.844940 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.845329 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.845334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.845337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.845509 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.845995 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.845999 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.846172 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.846299 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.846506 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 22:58:44.847103 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.833036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33334 (local address 127.8.204.189:42017)
0927 22:58:44.833230 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.833236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.833310 (+ 74us) server_negotiation.cc:407] Connection header received
0927 22:58:44.833552 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.833558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.833650 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.833767 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.834860 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.835686 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.837059 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.837475 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.837571 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.841032 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.841037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.841041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.841074 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.844252 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.845076 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.845080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.845082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.845209 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.845620 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.845623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.845625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.845882 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.845990 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.846430 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.846660 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:58:44.848160 31578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:44.849052 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.833718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.8.204.190:37649)
0927 22:58:44.833870 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.833874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.833889 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:58:44.834048 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.834051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.834103 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.834175 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.835579 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.836411 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.837881 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.838451 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.838543 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.842971 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.842975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.842978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.843010 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.846026 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.846802 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.846816 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.846820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.846976 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.847482 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.847487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.847491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.847879 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.848158 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.848518 (+ 360us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.848736 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:58:44.862465 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.848875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37718)
0927 22:58:44.849058 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.849075 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.849160 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.849781 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.849786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.850209 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.850389 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.850404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.851749 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.851753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.852773 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.852787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.853438 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.853446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.853661 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.854226 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.854242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.856635 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.859923 (+ 3288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.859927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.859929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.860251 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.860697 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.860701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.860703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.860823 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.861502 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.861505 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.861619 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.861833 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.862186 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":18}
I20260927 22:58:44.862728 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.849117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.8.204.188:39121)
0927 22:58:44.849266 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.849272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.849288 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:58:44.849601 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.849607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.849671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.849769 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.850540 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.851629 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.852911 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.853295 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.853450 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.856823 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.856830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.856838 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.856877 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.859769 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.860397 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.860406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.860412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.860557 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.860932 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.860938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.860962 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.861357 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.861523 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.861967 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.862374 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":55}
I20260927 22:58:44.865772 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.850631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.8.204.189:42017)
0927 22:58:44.850783 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.850787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.850802 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:58:44.850976 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.850980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.851033 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.851105 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.852435 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.853315 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.854565 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.855234 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.855408 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.859923 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.859929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.859933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.859970 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.863233 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.863861 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.863867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.863871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.864006 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.864423 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.864426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.864428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.864657 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.864858 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.865303 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.865476 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:58:44.868468 31578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: Bootstrap starting.
I20260927 22:58:44.870524 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:44.872153 31578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:44.873440 31578 log.cc:824] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:44.876072 31578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: No bootstrap required, opened a new log
I20260927 22:58:44.873847 31405 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "286ac7bd56934e43b86473315160c0e1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52206
I20260927 22:58:44.882897 31578 raft_consensus.cc:348] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:44.883219 31578 raft_consensus.cc:729] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40feb4628664cae80ab7839085de515, State: Initialized, Role: FOLLOWER
I20260927 22:58:44.883679 31578 consensus_queue.cc:260] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:44.884624 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:44.885022 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:44.885550 31578 sys_catalog.cc:564] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:58:44.894788 31595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:58:44.894923 31595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:58:44.892855 31511 master_runner.cc:186] Error getting master registration for 127.8.204.190:37649: OK, instance_id { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" instance_seqno: 1790549923795482 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:58:44.900746 31511 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:58:44.940495 31430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 4
Sep 27 22:58:44 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549923, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:58:44.954110 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.941299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37734)
0927 22:58:44.941486 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:44.941509 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:44.941638 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:44.941995 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:44.942002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:44.942439 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:44.942601 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.942610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.943736 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.943743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.944728 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:44.944740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.945541 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.945546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:44.945780 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.946382 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:44.946400 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:44.948571 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:44.951650 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.951657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.951660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.952050 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.952417 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:44.952422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:44.952425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:44.952562 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:44.953143 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:44.953149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:44.953318 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 22:58:44.953472 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:58:44.953730 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 22:58:44.954120 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:44.941379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.8.204.188:39121)
0927 22:58:44.941640 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:44.941647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:44.941668 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:58:44.941765 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:44.941770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:44.941851 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:44.941956 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:44.942752 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.943586 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.944871 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:44.945394 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:44.945526 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:44.948745 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:44.948750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:44.948754 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:44.948783 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:44.951528 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.952188 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.952192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.952194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.952305 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:44.952671 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:44.952677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:44.952680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:44.953016 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:44.953182 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:44.953556 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:58:44.953756 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 22:58:44.959853 31430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1: Bootstrap starting.
I20260927 22:58:44.963199 31430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:44.964509 31430 log.cc:824] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:44.967355 31430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1: No bootstrap required, opened a new log
I20260927 22:58:44.972460 31430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:44.972785 31430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286ac7bd56934e43b86473315160c0e1, State: Initialized, Role: FOLLOWER
I20260927 22:58:44.973253 31430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:44.974198 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:44.974596 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:44.975325 31430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:44.975423 31405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:58:44.975636 31405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:44.979030 31405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:44.980574 31405 leader_election.cc:288] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [CANDIDATE]: Term 1 election: Requested vote from peers 903cb17c2b374959b32dc11121e90769 (127.8.204.189:42017), b40feb4628664cae80ab7839085de515 (127.8.204.188:39121)
I20260927 22:58:44.982064 31553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286ac7bd56934e43b86473315160c0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b40feb4628664cae80ab7839085de515"
I20260927 22:58:44.982631 31553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:44.987360 31553 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286ac7bd56934e43b86473315160c0e1 in term 1.
I20260927 22:58:44.988286 31381 leader_election.cc:302] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286ac7bd56934e43b86473315160c0e1, b40feb4628664cae80ab7839085de515; no voters:
I20260927 22:58:44.988860 31598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:44.998525 31598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [term 1 LEADER]: Becoming Leader. State: Replica: 286ac7bd56934e43b86473315160c0e1, State: Running, Role: LEADER
I20260927 22:58:44.999312 31598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:45.008524 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286ac7bd56934e43b86473315160c0e1. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.009156 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:45.017769 31504 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } attempt: 5
I20260927 22:58:45.027807 31610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 22:58:45 dist-test-slave-40vx krb5kdc[31350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549924, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
W20260927 22:58:45.030453 31532 tablet.cc:2369] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:58:45.035017 31384 tablet.cc:2369] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:58:45.042776 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.018545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:39121 (local address 127.0.0.1:37746)
0927 22:58:45.019084 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:45.019109 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:45.019258 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:45.019985 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:45.019993 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:45.020702 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:45.020972 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:45.020986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.022961 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.022970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:45.024954 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:45.024974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.026607 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.026616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:45.026960 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.027702 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:45.027726 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:45.032577 (+ 4851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:45.038795 (+ 6218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:45.038804 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:45.038811 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:45.039593 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:45.040235 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:45.040240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:45.040243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:45.040447 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:45.041362 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:45.041371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:45.041589 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 22:58:45.041802 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 22:58:45.042212 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374}
I20260927 22:58:45.044286 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.019029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.8.204.188:39121)
0927 22:58:45.019206 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.019213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.019234 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:58:45.019488 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.019497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.019607 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.019734 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.021195 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.022733 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.025177 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.026182 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.026338 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.032908 (+ 6570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:45.032919 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:45.032927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:45.032986 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:45.038368 (+ 5382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.039823 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.039832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.039839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.040063 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.040610 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.040619 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.040625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.041103 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:45.041284 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.043421 (+ 2137us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.043766 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:58:45.052480 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: Bootstrap starting.
I20260927 22:58:45.057811 31504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:45.067728 31504 log.cc:824] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:45.078223 31434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.017478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.8.204.190:37649)
0927 22:58:45.017685 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.017693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.040390 (+ 22697us) server_negotiation.cc:407] Connection header received
0927 22:58:45.040820 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.040826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.040913 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.040985 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.044670 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.045825 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.052253 (+ 6428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.052981 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.053080 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.064387 (+ 11307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:45.064395 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:45.064400 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:45.064442 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:45.068580 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.071143 (+ 2563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.071149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.071153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.071371 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.072587 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.072593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.072609 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.072971 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:45.073212 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.077223 (+ 4011us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.077572 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:58:45.079908 31610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:45.101197 31504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: No bootstrap required, opened a new log
I20260927 22:58:45.109223 31553 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [term 1 FOLLOWER]: Refusing update from remote peer 286ac7bd56934e43b86473315160c0e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:45.109588 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.023966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.8.204.189:42017)
0927 22:58:45.024138 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.024144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.058799 (+ 34655us) server_negotiation.cc:407] Connection header received
0927 22:58:45.060157 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.060166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.060263 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.060348 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.064002 (+ 3654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.065483 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.074653 (+ 9170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.075714 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.075864 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.093840 (+ 17976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:45.093849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:45.093854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:45.093898 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:45.097615 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.104318 (+ 6703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.104327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.104331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.104491 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.105960 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.105968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.105971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.106360 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:45.106610 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.108958 (+ 2348us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.109206 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:58:45.110898 31598 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:45.111585 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.058987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.8.204.188:39121)
0927 22:58:45.059224 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.059231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.075955 (+ 16724us) server_negotiation.cc:407] Connection header received
0927 22:58:45.076669 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.076676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.076768 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.076848 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.083432 (+ 6584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.084572 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.087803 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.088546 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.088638 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.098903 (+ 10265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:45.098910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:45.098915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:45.098957 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:45.102834 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.105591 (+ 2757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.105597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.105600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.105763 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:45.107707 (+ 1944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:45.107712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:45.107716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:45.108067 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:45.108343 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.110954 (+ 2611us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.111194 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:58:45.129029 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286ac7bd56934e43b86473315160c0e1. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.129456 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:45.134882 31504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:45.135308 31504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 903cb17c2b374959b32dc11121e90769, State: Initialized, Role: FOLLOWER
I20260927 22:58:45.135810 31504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } }
I20260927 22:58:45.139438 31610 catalog_manager.cc:1347] Generated new cluster ID: 05e2dc40f1fb4b859600ae5e605d31d1
I20260927 22:58:45.139546 31610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:45.143913 31605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.144192 31605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:45.155447 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.155664 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:45.139511 31363 master_runner.cc:186] Error getting master registration for 127.8.204.189:42017: OK, instance_id { permanent_uuid: "903cb17c2b374959b32dc11121e90769" instance_seqno: 1790549924292525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:58:45.157841 31363 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:58:45.157989 31623 mvcc.cc:205] Tried to move back new op lower bound from 7334092493219659776 to 7334092492804837376. Current Snapshot: MvccSnapshot[applied={T|T < 7334092493219659776 or (T in {7334092493219659776})}]
I20260927 22:58:45.162582 31610 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:45.165288 31504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:45.165791 31479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "286ac7bd56934e43b86473315160c0e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "903cb17c2b374959b32dc11121e90769"
I20260927 22:58:45.166167 31610 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:45.167126 31478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:45.170898 31478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [term 1 FOLLOWER]: Refusing update from remote peer 286ac7bd56934e43b86473315160c0e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:45.175954 31598 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:45.177847 31625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.178357 31625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:45.179723 31479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286ac7bd56934e43b86473315160c0e1 in term 1.
I20260927 22:58:45.190848 31610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1: Generated new TSK 0
I20260927 22:58:45.192032 31610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:45.254523 31625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 286ac7bd56934e43b86473315160c0e1. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.254880 31625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:45.255513 31641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.255935 31641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:45.260830 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "286ac7bd56934e43b86473315160c0e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "286ac7bd56934e43b86473315160c0e1" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 37649 } } peers { permanent_uuid: "903cb17c2b374959b32dc11121e90769" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 42017 } } peers { permanent_uuid: "b40feb4628664cae80ab7839085de515" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 39121 } } }
I20260927 22:58:45.261183 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 286ac7bd56934e43b86473315160c0e1 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:45.294059 31645 catalog_manager.cc:1259] Loaded cluster ID: 05e2dc40f1fb4b859600ae5e605d31d1
I20260927 22:58:45.294253 31645 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: loading cluster ID for follower catalog manager: success
I20260927 22:58:45.298040 31645 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: acquiring CA information for follower catalog manager: success
I20260927 22:58:45.300586 31645 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 903cb17c2b374959b32dc11121e90769: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:58:45.355389 31434 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.330217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.8.204.190:37649)
0927 22:58:45.330404 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.330410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.350896 (+ 20486us) server_negotiation.cc:407] Connection header received
0927 22:58:45.353605 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.353612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.353760 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.353849 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.355085 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52226: BlockingRecv error: recv got EOF from 127.0.0.1:52226 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 22:58:45.358392 31508 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.336374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.8.204.189:42017)
0927 22:58:45.336565 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.336571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.356498 (+ 19927us) server_negotiation.cc:407] Connection header received
0927 22:58:45.356914 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.356921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.357029 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.357100 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.358176 (+ 1076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33372: BlockingRecv error: recv got EOF from 127.0.0.1:33372 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 22:58:45.360718 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.351496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.8.204.188:39121)
0927 22:58:45.351706 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.351712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.358998 (+ 7286us) server_negotiation.cc:407] Connection header received
0927 22:58:45.359419 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.359425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.359517 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.359585 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.360444 (+ 859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37764: BlockingRecv error: recv got EOF from 127.0.0.1:37764 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:58:45.411619 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.392029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.8.204.189:42017)
0927 22:58:45.392247 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.392254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.403912 (+ 11658us) server_negotiation.cc:407] Connection header received
0927 22:58:45.404206 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.404213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.404423 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.404575 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.405319 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.406465 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.408519 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.409204 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.409311 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.410076 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.410791 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.410918 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.411158 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.411257 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:58:45.457768 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.447797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37780 (local address 127.8.204.188:39121)
0927 22:58:45.447995 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.448002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.455819 (+ 7817us) server_negotiation.cc:407] Connection header received
0927 22:58:45.456152 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.456159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.456314 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.456397 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.457499 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37780: BlockingRecv error: recv got EOF from 127.0.0.1:37780 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:58:45.502846 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.488157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.8.204.189:42017)
0927 22:58:45.488400 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.488407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.495812 (+ 7405us) server_negotiation.cc:407] Connection header received
0927 22:58:45.496256 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.496263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.496385 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.496452 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.497106 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.498356 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.500456 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.501201 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.501319 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.501938 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.502112 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.502177 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.502427 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.502515 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 22:58:45.535779 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.533591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37784 (local address 127.8.204.188:39121)
0927 22:58:45.533803 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.533810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.534289 (+ 479us) server_negotiation.cc:407] Connection header received
0927 22:58:45.534497 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.534504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.534605 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.534723 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.535532 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37784: BlockingRecv error: recv got EOF from 127.0.0.1:37784 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:58:45.547973 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.541163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.8.204.189:42017)
0927 22:58:45.541340 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.541348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.541881 (+ 533us) server_negotiation.cc:407] Connection header received
0927 22:58:45.542050 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.542056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.542156 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.542258 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.542695 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.543929 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.545878 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.546554 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.546723 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.547127 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.547266 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.547381 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.547527 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.547607 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:58:45.559397 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.557161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.8.204.188:39121)
0927 22:58:45.557332 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.557337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.557879 (+ 542us) server_negotiation.cc:407] Connection header received
0927 22:58:45.558056 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.558062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.558164 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.558299 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.559150 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37794: BlockingRecv error: recv got EOF from 127.0.0.1:37794 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:58:45.582012 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.572773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.8.204.189:42017)
0927 22:58:45.572969 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.572976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.575581 (+ 2605us) server_negotiation.cc:407] Connection header received
0927 22:58:45.575814 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.575820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.575910 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.576014 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.576566 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.577821 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.579843 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.580593 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.580724 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.581212 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.581345 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.581438 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.581638 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.581733 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:58:45.595064 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.592831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.8.204.188:39121)
0927 22:58:45.593093 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.593099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.593626 (+ 527us) server_negotiation.cc:407] Connection header received
0927 22:58:45.593807 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.593813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.593923 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.594036 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.594848 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37808: BlockingRecv error: recv got EOF from 127.0.0.1:37808 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 22:58:45.613621 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.606832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.8.204.189:42017)
0927 22:58:45.607008 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.607014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.607533 (+ 519us) server_negotiation.cc:407] Connection header received
0927 22:58:45.607670 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.607676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.607774 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.607873 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.608322 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.609416 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.611395 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.612090 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.612229 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.612716 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.612900 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.612999 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.613177 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.613262 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:58:45.623624 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.621240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37810 (local address 127.8.204.188:39121)
0927 22:58:45.621432 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.621438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.621976 (+ 538us) server_negotiation.cc:407] Connection header received
0927 22:58:45.622144 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.622151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.622249 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.622383 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.623343 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37810: BlockingRecv error: recv got EOF from 127.0.0.1:37810 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:58:45.650826 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.643777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.8.204.189:42017)
0927 22:58:45.643990 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.643997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.644568 (+ 571us) server_negotiation.cc:407] Connection header received
0927 22:58:45.644795 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.644800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.644892 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.644991 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.645495 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.646707 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.648724 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.649415 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.649557 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.650066 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.650206 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.650301 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.650458 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.650550 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 22:58:45.665544 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.661019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.8.204.188:39121)
0927 22:58:45.661219 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.661225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.663904 (+ 2679us) server_negotiation.cc:407] Connection header received
0927 22:58:45.664084 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.664091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.664198 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.664311 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.665269 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37814: BlockingRecv error: recv got EOF from 127.0.0.1:37814 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:58:45.709069 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.701802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.8.204.189:42017)
0927 22:58:45.702027 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.702038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.702597 (+ 559us) server_negotiation.cc:407] Connection header received
0927 22:58:45.702804 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.702810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.702921 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.703016 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.703557 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.704719 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.706701 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.707408 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.707535 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.708160 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.708305 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.708401 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.708628 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.708728 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 22:58:45.719785 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.717489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.8.204.188:39121)
0927 22:58:45.717678 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.717684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.718236 (+ 552us) server_negotiation.cc:407] Connection header received
0927 22:58:45.718388 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.718394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.718490 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.718604 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.719543 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37818: BlockingRecv error: recv got EOF from 127.0.0.1:37818 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 22:58:45.792338 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.787098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.8.204.189:42017)
0927 22:58:45.787368 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.787376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.787740 (+ 364us) server_negotiation.cc:407] Connection header received
0927 22:58:45.787961 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.787967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.788106 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.788202 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.788674 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.789387 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.790675 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.791080 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.791213 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.791627 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.791744 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.791843 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.791986 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.792064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 22:58:45.800839 31581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:45.798689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.8.204.188:39121)
0927 22:58:45.798909 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.798916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.799354 (+ 438us) server_negotiation.cc:407] Connection header received
0927 22:58:45.799587 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.799592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.799679 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.799784 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:45.800634 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37820: BlockingRecv error: recv got EOF from 127.0.0.1:37820 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:58:45.896989 31595 catalog_manager.cc:1259] Loaded cluster ID: 05e2dc40f1fb4b859600ae5e605d31d1
I20260927 22:58:45.897087 31595 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: loading cluster ID for follower catalog manager: success
I20260927 22:58:45.899511 31595 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: acquiring CA information for follower catalog manager: success
I20260927 22:58:45.901157 31595 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b40feb4628664cae80ab7839085de515: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:58:45.940950 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.935750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.8.204.189:42017)
0927 22:58:45.936016 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.936024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.936297 (+ 273us) server_negotiation.cc:407] Connection header received
0927 22:58:45.936542 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.936549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.936639 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.936747 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.937167 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.937843 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.939142 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.939621 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.939754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.940129 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.940332 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.940449 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.940607 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.940683 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:58:45.953915 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.948310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37832 (local address 127.8.204.188:39121)
0927 22:58:45.948504 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.948512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.948807 (+ 295us) server_negotiation.cc:407] Connection header received
0927 22:58:45.949012 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.949016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.949161 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.949279 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.949681 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.950372 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.951723 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.952151 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.952280 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.952773 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.953327 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.953499 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.953549 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.953630 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 22:58:45.968911 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.959852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.8.204.190:37649)
0927 22:58:45.961846 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.961864 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.961880 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:58:45.961969 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.961973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.962118 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.962295 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.962984 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.964086 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.966465 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.966988 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.967124 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.967717 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.968245 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.968349 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.968480 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.968577 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1866,"thread_start_us":1774,"threads_started":1}
I20260927 22:58:45.969385 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.959886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.8.204.189:42017)
0927 22:58:45.960095 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.960104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.963037 (+ 2933us) server_negotiation.cc:407] Connection header received
0927 22:58:45.963173 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.963179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.963302 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.963393 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.964187 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.964997 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.967291 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.967889 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.967997 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.968595 (+ 598us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.968728 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.968818 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.969011 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.969093 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:58:45.970055 31581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:45.960874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.8.204.188:39121)
0927 22:58:45.961088 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:45.961096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:45.963609 (+ 2513us) server_negotiation.cc:407] Connection header received
0927 22:58:45.963800 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:45.963806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:45.963914 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:45.964026 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:45.964529 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.965639 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.967690 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:45.968241 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:45.968375 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:45.969231 (+ 856us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:45.969340 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:45.969417 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:45.969629 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 22:58:45.969697 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:58:45.973655 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31363
I20260927 22:58:45.974726 31425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:46.083966 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31363
I20260927 22:58:46.105031 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31437
I20260927 22:58:46.106060 31499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:46.213716 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31437
I20260927 22:58:46.233649 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31511
I20260927 22:58:46.234704 31573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:46.341164 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31511
2026-09-27T22:58:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4326 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:58:46 dist-test-slave-40vx krb5kdc[31757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:58:46 dist-test-slave-40vx krb5kdc[31757](info): set up 2 sockets
Sep 27 22:58:46 dist-test-slave-40vx krb5kdc[31757](info): commencing operation
krb5kdc: starting...
W20260927 22:58:47.443922 9010 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/test-user.keytab.
Sep 27 22:58:47 dist-test-slave-40vx krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:58:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:58:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.204.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
Entry for principal kudu/127.8.204.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab.
I20260927 22:58:47.693959 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39771
--webserver_interface=127.8.204.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:39771,127.8.204.189:39989,127.8.204.188:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:58:48.021799 31770 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:58:48.022210 31770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:48.022346 31770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:48.022485 31770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:48.029886 31770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:48.030004 31770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:48.030146 31770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:48.030238 31770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:48.030347 31770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:48.040120 31770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:39771,127.8.204.189:39989,127.8.204.188:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.190.keytab
--principal=kudu/127.8.204.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.190:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.204.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:48.041414 31770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:48.042707 31770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:48.054358 31776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:48.054437 31778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:48.054464 31775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:48.055521 31770 server_base.cc:1034] running on GCE node
I20260927 22:58:48.056198 31770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:48.057479 31770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:48.058730 31770 hybrid_clock.cc:648] HybridClock initialized: now 1790549928058680 us; error 47 us; skew 500 ppm
I20260927 22:58:48.059048 31770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:48 dist-test-slave-40vx krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:48.062156 31770 init.cc:377] Logged in from keytab as kudu/127.8.204.190@KRBTEST.COM (short username kudu)
I20260927 22:58:48.063534 31770 webserver.cc:460] Webserver started at http://127.8.204.190:33511/ using document root <none> and password file <none>
I20260927 22:58:48.064188 31770 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:48.064311 31770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:48.064604 31770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:48.067426 31770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data/instance:
uuid: "4c86f4cd8e5f497f88c96496978c0186"
format_stamp: "Formatted at 2026-09-27 22:58:48 on dist-test-slave-40vx"
server_key: "5bfaadc36167ceb70626c0dbccdcd6cc"
server_key_iv: "51ededb05e9b79597731f35802cb63b8"
server_key_version: "encryptionkey@0"
I20260927 22:58:48.068181 31770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal/instance:
uuid: "4c86f4cd8e5f497f88c96496978c0186"
format_stamp: "Formatted at 2026-09-27 22:58:48 on dist-test-slave-40vx"
server_key: "5bfaadc36167ceb70626c0dbccdcd6cc"
server_key_iv: "51ededb05e9b79597731f35802cb63b8"
server_key_version: "encryptionkey@0"
I20260927 22:58:48.072994 31770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:58:48.076108 31785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:48.077319 31770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:58:48.077497 31770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal
uuid: "4c86f4cd8e5f497f88c96496978c0186"
format_stamp: "Formatted at 2026-09-27 22:58:48 on dist-test-slave-40vx"
server_key: "5bfaadc36167ceb70626c0dbccdcd6cc"
server_key_iv: "51ededb05e9b79597731f35802cb63b8"
server_key_version: "encryptionkey@0"
I20260927 22:58:48.077694 31770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:48.093391 31770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:48.097671 31770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:48.097959 31770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:48.116071 31770 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.190:39771
I20260927 22:58:48.116096 31836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.190:39771 every 8 connection(s)
I20260927 22:58:48.117345 31770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/data/info.pb
I20260927 22:58:48.117937 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31770
I20260927 22:58:48.118170 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-0/wal/instance
I20260927 22:58:48.118785 9010 external_mini_cluster.cc:1428] Setting key 71d087e94b4de49d2c0ceaf1e6f6fce6
I20260927 22:58:48.121310 31837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:48.128788 31837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.204.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.189@KRBTEST.COM" created.
Sep 27 22:58:48 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:48.155428 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.132566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39771 (local address 127.0.0.1:50602)
0927 22:58:48.133434 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.133490 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:48.136370 (+ 2880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:48.137205 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:48.137217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:48.138053 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:48.138577 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.138603 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.140174 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.140180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.142194 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.142212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.143179 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.143190 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.143430 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.144098 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:48.144130 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:48.148116 (+ 3986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:48.151252 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.151265 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.151296 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.151697 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.152087 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.152093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.152099 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.152255 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.152977 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:48.152984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:48.153187 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 22:58:48.153417 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 22:58:48.154059 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":200,"thread_start_us":360,"threads_started":1}
I20260927 22:58:48.155423 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.132566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.8.204.190:39771)
0927 22:58:48.134431 (+ 1865us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:48.134437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:48.134477 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:58:48.136735 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:48.136741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:48.136968 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:48.137115 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:48.138792 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.140017 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.142341 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.142875 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.143037 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.148356 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:48.148364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:48.148375 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:48.148411 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:48.151113 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.151825 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.151834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.151839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.151979 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.152365 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.152376 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.152379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.152838 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:48.153047 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:48.153556 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:58:48.153948 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":1686,"spinlock_wait_cycles":23296,"thread_start_us":1457,"threads_started":1}
I20260927 22:58:48.158514 31837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:48.159762 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.159347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54854)
0927 22:58:48.159526 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.159646 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 22:58:48.160053 31786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
I20260927 22:58:48.161600 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.161259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54868)
0927 22:58:48.161409 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.161471 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 22:58:48.161811 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:39989: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab.
Entry for principal kudu/127.8.204.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab.
I20260927 22:58:48.178385 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:39989
--webserver_interface=127.8.204.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab
--principal=kudu/127.8.204.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:39771,127.8.204.189:39989,127.8.204.188:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:58:48.211130 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } attempt: 1
I20260927 22:58:48.212436 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.211868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54884)
0927 22:58:48.212114 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.212261 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 22:58:48.213465 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.213057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54894)
0927 22:58:48.213218 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.213313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:58:48.213682 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:39989: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
I20260927 22:58:48.281930 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } attempt: 2
I20260927 22:58:48.283197 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.282639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54902)
0927 22:58:48.282911 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.283025 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:58:48.284288 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.283860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54912)
0927 22:58:48.284032 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.284113 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:58:48.284529 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:39989: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
I20260927 22:58:48.375794 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } attempt: 3
I20260927 22:58:48.377081 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.376521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54926)
0927 22:58:48.376778 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.376914 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:58:48.378159 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.377760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54930)
0927 22:58:48.377907 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.377994 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:58:48.378345 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:39989: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
W20260927 22:58:48.507984 31844 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:58:48.508309 31844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:48.508446 31844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:48.508558 31844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:48.515703 31844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:48.515785 31844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:48.515910 31844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:48.515995 31844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:48.516081 31844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:48.521620 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } attempt: 4
I20260927 22:58:48.522832 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.522311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54934)
0927 22:58:48.522615 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.522704 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 22:58:48.523922 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.523593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54940)
0927 22:58:48.523726 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.523806 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:58:48.524156 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.189:39989: Network error: Client connection negotiation failed: client connection to 127.8.204.189:39989: connect: Connection refused (error 111)
I20260927 22:58:48.526564 31844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:39771,127.8.204.189:39989,127.8.204.188:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.189.keytab
--principal=kudu/127.8.204.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.189:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.204.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:48.527958 31844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:48.529265 31844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:58:48.541431 31852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:48.541481 31849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:48.541433 31850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:48.542301 31844 server_base.cc:1034] running on GCE node
I20260927 22:58:48.543023 31844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:48.544423 31844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:48.545641 31844 hybrid_clock.cc:648] HybridClock initialized: now 1790549928545562 us; error 60 us; skew 500 ppm
I20260927 22:58:48.545979 31844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:48 dist-test-slave-40vx krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:48.549247 31844 init.cc:377] Logged in from keytab as kudu/127.8.204.189@KRBTEST.COM (short username kudu)
I20260927 22:58:48.550686 31844 webserver.cc:460] Webserver started at http://127.8.204.189:35445/ using document root <none> and password file <none>
I20260927 22:58:48.551438 31844 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:48.551568 31844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:48.551882 31844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:48.554790 31844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data/instance:
uuid: "222fd8279e1340de8073de6d4fb10870"
format_stamp: "Formatted at 2026-09-27 22:58:48 on dist-test-slave-40vx"
server_key: "1ae2a671c2f50b4a6af36510c3c40057"
server_key_iv: "0101e1d30cd6662ed91a1f22029998a1"
server_key_version: "encryptionkey@0"
I20260927 22:58:48.555634 31844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal/instance:
uuid: "222fd8279e1340de8073de6d4fb10870"
format_stamp: "Formatted at 2026-09-27 22:58:48 on dist-test-slave-40vx"
server_key: "1ae2a671c2f50b4a6af36510c3c40057"
server_key_iv: "0101e1d30cd6662ed91a1f22029998a1"
server_key_version: "encryptionkey@0"
I20260927 22:58:48.560559 31844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:58:48.563983 31859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:48.565405 31844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:48.565593 31844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal
uuid: "222fd8279e1340de8073de6d4fb10870"
format_stamp: "Formatted at 2026-09-27 22:58:48 on dist-test-slave-40vx"
server_key: "1ae2a671c2f50b4a6af36510c3c40057"
server_key_iv: "0101e1d30cd6662ed91a1f22029998a1"
server_key_version: "encryptionkey@0"
I20260927 22:58:48.565822 31844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:48.579129 31844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:48.583490 31844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:48.583753 31844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:48.601863 31844 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.189:39989
I20260927 22:58:48.601881 31910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.189:39989 every 8 connection(s)
I20260927 22:58:48.603186 31844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/data/info.pb
I20260927 22:58:48.605756 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31844
I20260927 22:58:48.605962 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-1/wal/instance
I20260927 22:58:48.606381 9010 external_mini_cluster.cc:1428] Setting key 30c88c5be8df216040d94f3ae9ee2a7d
I20260927 22:58:48.606896 31911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:48.613121 31911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:48 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.204.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.204.188@KRBTEST.COM" created.
I20260927 22:58:48.633934 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.615787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50604 (local address 127.8.204.190:39771)
0927 22:58:48.616026 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:48.616036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:48.617192 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 22:58:48.619030 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:48.619038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:48.619131 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:48.619252 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:48.620779 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.621944 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.623458 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.624295 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.624427 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.627782 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:48.627789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:48.627797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:48.627841 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:48.630860 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.631628 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.631633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.631635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.631762 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.632263 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.632267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.632269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.632528 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:48.632714 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:48.633377 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 22:58:48.633587 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:58:48.634888 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.616242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39771 (local address 127.0.0.1:50604)
0927 22:58:48.617006 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.617057 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:48.618754 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:48.619570 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:48.619577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:48.620069 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:48.620595 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.620611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.622106 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.622110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.623245 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.623257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.624520 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.624524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.624709 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.625376 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:48.625402 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:48.627569 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:48.630988 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.630994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.631004 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.631517 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.631941 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.631945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.631948 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.632155 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.632685 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:48.632693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:48.633020 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 22:58:48.633272 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 22:58:48.633748 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":343,"threads_started":1}
I20260927 22:58:48.636533 31911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:48 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:58:48.653574 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.637322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54952)
0927 22:58:48.637529 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.637549 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:48.637653 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:48.638575 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:48.638583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:48.639094 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:48.639312 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.639325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.641425 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.641429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.642261 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.642271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.643483 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.643487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.643622 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.644222 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:48.644238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:48.646364 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:48.650675 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.650679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.650682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.651128 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.651672 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.651678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.651684 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.651844 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.652574 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:48.652579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:48.652721 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 22:58:48.652869 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:58:48.653166 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":2,"spinlock_wait_cycles":15232}
I20260927 22:58:48.653659 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.637538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.8.204.189:39989)
0927 22:58:48.638184 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:48.638189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:48.638213 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:58:48.638252 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:48.638257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:48.638416 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:48.638522 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:48.639484 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.641294 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.642396 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.643356 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.643492 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.646587 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:48.646603 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:48.646622 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:48.646662 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:48.650535 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.651323 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.651332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.651336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.651540 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.651955 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.651960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.651963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.652438 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:48.652629 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:48.653028 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 22:58:48.653274 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":504,"thread_start_us":379,"threads_started":1}
I20260927 22:58:48.656180 31911 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:48.657286 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.656860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57540)
0927 22:58:48.657046 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.657143 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260927 22:58:48.657526 31860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:48.658982 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.658617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57542)
0927 22:58:48.658756 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.658844 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:58:48.659211 31911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.204.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab.
Entry for principal kudu/127.8.204.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab.
I20260927 22:58:48.667757 9010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
/tmp/dist-test-taskoq4u9H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:42393
--webserver_interface=127.8.204.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab
--principal=kudu/127.8.204.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.204.148:32871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.204.190:39771,127.8.204.189:39989,127.8.204.188:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:58:48.708638 31911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 1
I20260927 22:58:48.709847 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.709384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57546)
0927 22:58:48.709575 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.709689 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:58:48.710876 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.710559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57554)
0927 22:58:48.710691 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.710764 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 22:58:48.711158 31911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:48.779373 31911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 2
I20260927 22:58:48.780544 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.780102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57564)
0927 22:58:48.780293 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.780398 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:58:48.781595 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.781251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57572)
0927 22:58:48.781397 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.781476 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:58:48.781785 31911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:48.823393 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } attempt: 5
Sep 27 22:58:48 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:58:48.843098 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.824390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.8.204.189:39989)
0927 22:58:48.824619 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:48.824625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:48.824650 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:58:48.824890 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:48.824894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:48.824957 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:48.825064 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:48.825895 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.826584 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.827890 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.828335 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.828469 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.837780 (+ 9311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:48.837785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:48.837789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:48.837822 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:48.840538 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.841205 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.841211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.841215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.841352 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:48.841710 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:48.841717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:48.841720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:48.842009 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:48.842160 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:48.842544 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 22:58:48.842762 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:58:48.843101 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.824253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54968)
0927 22:58:48.824482 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.824504 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:48.824681 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:48.825101 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:48.825107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:48.825547 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:48.825733 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.825742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.826713 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.826721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.827732 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:48.827752 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:48.828489 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:48.828496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:48.828713 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:48.834918 (+ 6205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:48.834942 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:48.837535 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:48.840687 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.840692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.840696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.841082 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.841464 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:48.841469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:48.841473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:48.841606 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:48.842125 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:48.842129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:48.842281 (+ 152us) client_negotiation.cc:769] Sending connection context
0927 22:58:48.842446 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:58:48.842741 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:58:48.844465 31837 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:48.845405 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.845039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57584)
0927 22:58:48.845197 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.845277 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 22:58:48.846375 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.846067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57596)
0927 22:58:48.846208 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.846281 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 22:58:48.846597 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:48.872980 31911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 3
I20260927 22:58:48.874105 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.873666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57608)
0927 22:58:48.873859 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.873954 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:58:48.875165 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.874748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57612)
0927 22:58:48.874973 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.875048 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260927 22:58:48.875392 31911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:48.897816 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 1
I20260927 22:58:48.899115 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.898520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57616)
0927 22:58:48.898776 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.898895 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:58:48.900213 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.899854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57626)
0927 22:58:48.900004 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.900089 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:58:48.900447 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:48.968856 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 2
I20260927 22:58:48.970186 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.969583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57636)
0927 22:58:48.969859 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.969975 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:58:48.971503 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:48.971056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57652)
0927 22:58:48.971242 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:48.971368 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:58:48.971746 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
W20260927 22:58:48.996208 31918 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:58:48.996541 31918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:58:48.996654 31918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:58:48.996784 31918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:58:49.003924 31918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:58:49.004002 31918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:58:49.004103 31918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:58:49.004190 31918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:58:49.004303 31918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:58:49.014088 31918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.204.148:32871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.204.190:39771,127.8.204.189:39989,127.8.204.188:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/krb5kdc/kudu_127.8.204.188.keytab
--principal=kudu/127.8.204.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.204.188:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.204.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 21:53:17 UTC on ccycloud.quasar-colwau.root.comops.site
build id 4676386
ASAN enabled
I20260927 22:58:49.015390 31918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:58:49.016687 31918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 22:58:49.018623 31911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 4
I20260927 22:58:49.019938 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.019337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57654)
0927 22:58:49.019607 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.019734 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:58:49.021073 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.020739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57658)
0927 22:58:49.020902 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.020973 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 22:58:49.021401 31911 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
W20260927 22:58:49.029035 31924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:49.029060 31923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:58:49.029033 31926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:58:49.030048 31918 server_base.cc:1034] running on GCE node
I20260927 22:58:49.030808 31918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:58:49.032191 31918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:58:49.033413 31918 hybrid_clock.cc:648] HybridClock initialized: now 1790549929033349 us; error 31 us; skew 500 ppm
I20260927 22:58:49.033746 31918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:58:49.036742 31918 init.cc:377] Logged in from keytab as kudu/127.8.204.188@KRBTEST.COM (short username kudu)
I20260927 22:58:49.038070 31918 webserver.cc:460] Webserver started at http://127.8.204.188:37281/ using document root <none> and password file <none>
I20260927 22:58:49.038734 31918 fs_manager.cc:359] Metadata directory not provided
I20260927 22:58:49.038861 31918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:58:49.039165 31918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:58:49.042071 31918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data/instance:
uuid: "61e059f90cd645e68807fb11ca1da57c"
format_stamp: "Formatted at 2026-09-27 22:58:49 on dist-test-slave-40vx"
server_key: "3841b164e81a33ddc0d9df1dfd4427cd"
server_key_iv: "5f3538bebcbb17c27ccadcd7c9980cd4"
server_key_version: "encryptionkey@0"
I20260927 22:58:49.042821 31918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal/instance:
uuid: "61e059f90cd645e68807fb11ca1da57c"
format_stamp: "Formatted at 2026-09-27 22:58:49 on dist-test-slave-40vx"
server_key: "3841b164e81a33ddc0d9df1dfd4427cd"
server_key_iv: "5f3538bebcbb17c27ccadcd7c9980cd4"
server_key_version: "encryptionkey@0"
I20260927 22:58:49.047657 31918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260927 22:58:49.051179 31933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:58:49.052510 31918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 22:58:49.052685 31918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data,/tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal
uuid: "61e059f90cd645e68807fb11ca1da57c"
format_stamp: "Formatted at 2026-09-27 22:58:49 on dist-test-slave-40vx"
server_key: "3841b164e81a33ddc0d9df1dfd4427cd"
server_key_iv: "5f3538bebcbb17c27ccadcd7c9980cd4"
server_key_version: "encryptionkey@0"
I20260927 22:58:49.052896 31918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:58:49.077973 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 3
I20260927 22:58:49.079043 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.078582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57662)
0927 22:58:49.078811 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.078890 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:58:49.079986 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.079656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57668)
0927 22:58:49.079806 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.079879 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 22:58:49.080194 31837 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.204.188:42393: Network error: Client connection negotiation failed: client connection to 127.8.204.188:42393: connect: Connection refused (error 111)
I20260927 22:58:49.080889 31918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:58:49.084887 31918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:58:49.085155 31918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:58:49.103222 31918 rpc_server.cc:307] RPC server started. Bound to: 127.8.204.188:42393
I20260927 22:58:49.103241 31984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.204.188:42393 every 8 connection(s)
I20260927 22:58:49.104615 31918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/data/info.pb
I20260927 22:58:49.108162 31985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:58:49.111732 9010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu as pid 31918
I20260927 22:58:49.111922 9010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskoq4u9H/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790549572925352-9010-0/minicluster-data/master-2/wal/instance
I20260927 22:58:49.112347 9010 external_mini_cluster.cc:1428] Setting key 126b9b4ec23019f7eaf3f537d76e0de7
I20260927 22:58:49.115099 31985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:58:49.135857 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.119710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.8.204.190:39771)
0927 22:58:49.119917 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.119923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.120632 (+ 709us) server_negotiation.cc:407] Connection header received
0927 22:58:49.120909 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.120919 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.121000 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.121105 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.122620 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.123369 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.125008 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.125683 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.125817 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.129662 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.129671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.129677 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.129716 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.132364 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.133319 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.133323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.133326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.133449 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.133868 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.133873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.133876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.134216 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.134371 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.135222 (+ 851us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.135496 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:58:49.136880 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.119641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.190:39771 (local address 127.0.0.1:50620)
0927 22:58:49.120487 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.120542 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:49.120749 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:49.121191 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:49.121200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:49.122002 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:49.122392 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.122419 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.123524 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.123530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.124855 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.124869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.125912 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.125920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.126134 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.126660 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:49.126686 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:49.129458 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:49.132493 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.132503 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.132512 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.133181 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.133549 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.133555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.133562 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.133760 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.134348 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:49.134356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:49.134740 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 22:58:49.135139 (+ 399us) client_negotiation.cc:240] Negotiation successful
0927 22:58:49.135648 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":9,"spinlock_wait_cycles":44288,"thread_start_us":434,"threads_started":1}
I20260927 22:58:49.137207 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.114094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.8.204.188:42393)
0927 22:58:49.114860 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.114871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.114909 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:58:49.117722 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.117750 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.118243 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.118631 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.120342 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.122420 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.124203 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.125125 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.125362 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.130078 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.130085 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.130095 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.130121 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.133532 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.134263 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.134268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.134270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.134447 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.134820 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.134826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.134829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.135214 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.135544 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.136611 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.136793 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":562,"thread_start_us":335,"threads_started":1}
I20260927 22:58:49.138499 31985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.190@KRBTEST.COM
I20260927 22:58:49.151803 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.139150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.189:39989 (local address 127.0.0.1:54972)
0927 22:58:49.139361 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.139376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:49.139506 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:49.139887 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:49.139891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:49.140353 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:49.140491 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.140499 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.141445 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.141452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.142321 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.142331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.143016 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.143022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.143197 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.143786 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:49.143803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:49.145921 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:49.149269 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.149276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.149280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.149780 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.150159 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.150165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.150169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.150308 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.150821 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:49.150827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:49.150991 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 22:58:49.151142 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 22:58:49.151453 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:58:49.151829 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.139249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54972 (local address 127.8.204.189:39989)
0927 22:58:49.139477 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.139483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.139502 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:58:49.139698 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.139703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.139767 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.139870 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.140630 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.141305 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.142465 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.142867 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.142998 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.146110 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.146115 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.146119 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.146150 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.149142 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.149916 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.149921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.149924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.150051 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.150446 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.150450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.150452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.150703 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.150858 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.151241 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.151477 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 22:58:49.152912 31985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:58:49.153347 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.140184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.8.204.190:39771)
0927 22:58:49.140379 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.140384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.140401 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:58:49.140546 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.140551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.140619 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.140712 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.141778 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.142487 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.143717 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.144130 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.144253 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.148131 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.148136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.148139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.148168 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.150793 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.151400 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.151409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.151413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.151534 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.151922 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.151928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.151931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.152241 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.152532 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.152793 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.153005 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549929, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.188@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.204.189@KRBTEST.COM
I20260927 22:58:49.165719 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.153565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57688)
0927 22:58:49.153702 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.153719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:49.153818 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:49.154384 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:49.154387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:49.154796 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:49.154982 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.154994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.156045 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.156050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.157101 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.157112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.157758 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.157763 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.157982 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.158491 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:49.158507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:49.160579 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:49.163336 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.163341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.163345 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.163835 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.164207 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.164214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.164219 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.164373 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.165034 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:49.165038 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:49.165152 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 22:58:49.165265 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 22:58:49.165461 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 22:58:49.166034 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.153955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57688 (local address 127.8.204.188:42393)
0927 22:58:49.154127 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.154133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.154151 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:58:49.154198 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.154201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.154249 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.154354 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.155122 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.155937 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.157234 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.157627 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.157758 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.160745 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.160750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.160753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.160778 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.163158 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.163956 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.163964 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.163968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.164083 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.164538 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.164544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.164547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.164906 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.165054 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.165397 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.165638 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260927 22:58:49.168660 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.154929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54978 (local address 127.8.204.189:39989)
0927 22:58:49.155085 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.155090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.155105 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:58:49.155219 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.155224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.155338 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.155426 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.156557 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.157389 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.158498 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.158905 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.159037 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.162945 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.162949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.162952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.162979 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.165991 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.166699 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.166707 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.166710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.166850 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.167243 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.167247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.167250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.167631 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.167920 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.168068 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.168339 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:58:49.171223 31985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: Bootstrap starting.
I20260927 22:58:49.175257 31985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:49.176221 9010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:58:49.176643 31985 log.cc:824] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:49.179510 31985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: No bootstrap required, opened a new log
I20260927 22:58:49.178959 31812 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4c86f4cd8e5f497f88c96496978c0186"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50624
I20260927 22:58:49.186522 31985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.186847 31985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61e059f90cd645e68807fb11ca1da57c, State: Initialized, Role: FOLLOWER
I20260927 22:58:49.187342 31985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.188319 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.188699 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.189225 31985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:58:49.198216 32002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:58:49.198338 32002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:58:49.196267 31918 master_runner.cc:186] Error getting master registration for 127.8.204.190:39771: OK, instance_id { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" instance_seqno: 1790549928111080 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:58:49.207327 31918 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:58:49.257395 31837 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 4
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.190@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:58:49.274710 31840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.258154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57702)
0927 22:58:49.258387 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.258411 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:49.258546 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:49.259153 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:49.259160 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:49.259749 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:49.259984 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.259998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.261575 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.261583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.263126 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.263140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.264202 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.264208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.264417 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.265120 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:49.265145 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:49.267849 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:49.271982 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.271990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.271996 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.272491 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.272871 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.272875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.272879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.273040 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.273707 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:49.273711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:49.273904 (+ 193us) client_negotiation.cc:769] Sending connection context
0927 22:58:49.274048 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:58:49.274317 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:58:49.274788 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.258207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.8.204.188:42393)
0927 22:58:49.258792 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.258798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.258817 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:58:49.258883 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.258888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.258970 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.259083 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.260158 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.261387 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.263315 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.264027 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.264157 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.268054 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.268061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.268066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.268106 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.271837 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.272607 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.272613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.272617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.272759 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.273152 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.273157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.273161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.273589 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.273763 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.274169 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.274405 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440}
I20260927 22:58:49.280740 31837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186: Bootstrap starting.
I20260927 22:58:49.284719 31837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:49.285856 31837 log.cc:824] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:49.288641 31837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186: No bootstrap required, opened a new log
I20260927 22:58:49.294128 31837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.294441 31837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c86f4cd8e5f497f88c96496978c0186, State: Initialized, Role: FOLLOWER
I20260927 22:58:49.294863 31837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.295817 32005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.296168 32005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.296883 31837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:49.297006 31812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:58:49.297211 31812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:49.300869 31812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.302382 31812 leader_election.cc:288] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [CANDIDATE]: Term 1 election: Requested vote from peers 222fd8279e1340de8073de6d4fb10870 (127.8.204.189:39989), 61e059f90cd645e68807fb11ca1da57c (127.8.204.188:42393)
I20260927 22:58:49.303730 31960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c86f4cd8e5f497f88c96496978c0186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "61e059f90cd645e68807fb11ca1da57c"
I20260927 22:58:49.304222 31960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:49.307760 31960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c86f4cd8e5f497f88c96496978c0186 in term 1.
I20260927 22:58:49.308467 31786 leader_election.cc:302] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c86f4cd8e5f497f88c96496978c0186, 61e059f90cd645e68807fb11ca1da57c; no voters:
I20260927 22:58:49.309046 32005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:58:49.311177 32005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [term 1 LEADER]: Becoming Leader. State: Replica: 4c86f4cd8e5f497f88c96496978c0186, State: Running, Role: LEADER
I20260927 22:58:49.311834 32005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.320633 31911 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } attempt: 5
I20260927 22:58:49.330770 32016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c86f4cd8e5f497f88c96496978c0186. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.331172 32016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: This master's current role is: LEADER
Sep 27 22:58:49 dist-test-slave-40vx krb5kdc[31757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790549928, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.204.189@KRBTEST.COM for kudu/127.8.204.188@KRBTEST.COM
I20260927 22:58:49.332738 32023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 22:58:49.337114 31939 tablet.cc:2369] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:58:49.341545 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.321258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.204.188:42393 (local address 127.0.0.1:57712)
0927 22:58:49.321639 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 22:58:49.321660 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:58:49.321778 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:58:49.322826 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:58:49.322834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:58:49.323534 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:58:49.323817 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.323835 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.325619 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.325627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.327289 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:58:49.327308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.328818 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.328826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:58:49.329115 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.329845 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:58:49.329867 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:58:49.333291 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:58:49.337855 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.337859 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.337864 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.338524 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.339061 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:58:49.339068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:58:49.339074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:58:49.339306 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:58:49.340190 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:58:49.340198 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:58:49.340432 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 22:58:49.340630 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:58:49.341015 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260927 22:58:49.342687 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.322031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.8.204.188:42393)
0927 22:58:49.322237 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.322246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.322270 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:58:49.322344 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.322353 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.322451 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.322568 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.324053 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.325421 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.327488 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.328412 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.328557 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.333544 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.333554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.333562 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.333612 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.337665 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.338702 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.338710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.338714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.338902 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.339460 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.339467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.339473 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.339892 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.340104 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.341907 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.342206 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:58:49.349709 31911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: Bootstrap starting.
W20260927 22:58:49.350770 31791 tablet.cc:2369] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:58:49.354158 32023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:58:49.354658 31911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: Neither blocks nor log segments found. Creating new log.
I20260927 22:58:49.363694 31911 log.cc:824] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: Log is configured to *not* fsync() on all Append() calls
I20260927 22:58:49.379485 31960 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [term 1 FOLLOWER]: Refusing update from remote peer 4c86f4cd8e5f497f88c96496978c0186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:49.380781 32005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:49.395643 31911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: No bootstrap required, opened a new log
I20260927 22:58:49.399614 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c86f4cd8e5f497f88c96496978c0186. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.399945 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.402470 32026 mvcc.cc:205] Tried to move back new op lower bound from 7334092510717792256 to 7334092510466945024. Current Snapshot: MvccSnapshot[applied={T|T < 7334092510717792256}]
I20260927 22:58:49.404618 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.404865 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.404970 32029 mvcc.cc:205] Tried to move back new op lower bound from 7334092510717792256 to 7334092510466945024. Current Snapshot: MvccSnapshot[applied={T|T < 7334092510717792256}]
I20260927 22:58:49.408722 31915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.322463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54980 (local address 127.8.204.189:39989)
0927 22:58:49.322680 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.322688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.355399 (+ 32711us) server_negotiation.cc:407] Connection header received
0927 22:58:49.355771 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.355779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.355889 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.355972 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.367414 (+ 11442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.368952 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.380184 (+ 11232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.381105 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.381228 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.390636 (+ 9408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.390643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.390647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.390687 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.394546 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.405475 (+ 10929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.405482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.405488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.405671 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.406285 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.406292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.406296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.406873 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.407315 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.407877 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.408168 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:58:49.409850 32005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.412161 32023 catalog_manager.cc:1347] Generated new cluster ID: 5a703d95238f47d3baa21c17310080be
I20260927 22:58:49.412259 32023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:58:49.413283 31911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.413632 31911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222fd8279e1340de8073de6d4fb10870, State: Initialized, Role: FOLLOWER
I20260927 22:58:49.414114 31911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } }
I20260927 22:58:49.415354 32005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: This master's current role is: LEADER
I20260927 22:58:49.415725 32030 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:58:49.421314 31841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.326228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.8.204.190:39771)
0927 22:58:49.326406 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.326411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.357219 (+ 30808us) server_negotiation.cc:407] Connection header received
0927 22:58:49.359726 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.359736 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.359857 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.359949 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.372099 (+ 12150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.373667 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.376382 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.377380 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.377512 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.401890 (+ 24378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.401897 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.401902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.401947 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.417520 (+ 15573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.418258 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.418263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.418267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.418402 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.418767 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.418771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.418775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.419137 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.419417 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.420622 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.420881 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:58:49.414986 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.435392 32023 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:58:49.435590 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.437446 32023 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:58:49.438180 31988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:49.347460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57718 (local address 127.8.204.188:42393)
0927 22:58:49.351512 (+ 4052us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.351520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.387791 (+ 36271us) server_negotiation.cc:407] Connection header received
0927 22:58:49.388037 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.388043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.388152 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.388224 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.400948 (+ 12724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.402013 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.405384 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:49.406095 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:49.406214 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:49.424542 (+ 18328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:58:49.424550 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:58:49.424555 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:58:49.424598 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:58:49.428205 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.435366 (+ 7161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.435373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.435378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.435520 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:58:49.436002 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:58:49.436007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:58:49.436011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:58:49.436373 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:58:49.436646 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:49.437532 (+ 886us) server_negotiation.cc:299] Negotiation successful
0927 22:58:49.437774 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3860}
I20260927 22:58:49.420877 31770 master_runner.cc:186] Error getting master registration for 127.8.204.189:39989: OK, instance_id { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" instance_seqno: 1790549928596740 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:58:49.445633 31770 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:58:49.449301 31911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:58:49.449378 31886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c86f4cd8e5f497f88c96496978c0186" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "222fd8279e1340de8073de6d4fb10870"
I20260927 22:58:49.449844 31886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:58:49.458365 31886 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c86f4cd8e5f497f88c96496978c0186 in term 1.
I20260927 22:58:49.459159 32023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186: Generated new TSK 0
I20260927 22:58:49.459327 31885 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [term 1 FOLLOWER]: Refusing update from remote peer 4c86f4cd8e5f497f88c96496978c0186: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:58:49.460214 32023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:58:49.461465 32016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [LEADER]: Connected to new peer: Peer: permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:58:49.512137 32044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.512568 32044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.513028 32034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c86f4cd8e5f497f88c96496978c0186. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.513300 32034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:58:49.527029 32052 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260927 22:58:49.558120 31841 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.549263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.8.204.190:39771)
0927 22:58:49.549496 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.549502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.556158 (+ 6656us) server_negotiation.cc:407] Connection header received
0927 22:58:49.556437 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.556444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.556525 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.556612 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.557864 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58988: BlockingRecv error: recv got EOF from 127.0.0.1:58988 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 22:58:49.561563 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.549789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34870 (local address 127.8.204.189:39989)
0927 22:58:49.550039 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.550048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.559833 (+ 9785us) server_negotiation.cc:407] Connection header received
0927 22:58:49.560091 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.560097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.560178 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.560247 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.561353 (+ 1106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34870: BlockingRecv error: recv got EOF from 127.0.0.1:34870 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 22:58:49.561865 31988 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.556299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59628 (local address 127.8.204.188:42393)
0927 22:58:49.556488 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.556497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.560201 (+ 3704us) server_negotiation.cc:407] Connection header received
0927 22:58:49.560442 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.560450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.560532 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.560602 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.561624 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59628: BlockingRecv error: recv got EOF from 127.0.0.1:59628 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 22:58:49.573148 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.568312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34876 (local address 127.8.204.189:39989)
0927 22:58:49.568486 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.568494 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.571735 (+ 3241us) server_negotiation.cc:407] Connection header received
0927 22:58:49.571936 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.571942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.572068 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.572173 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.572908 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34876: BlockingRecv error: recv got EOF from 127.0.0.1:34876 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 22:58:49.598950 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.591618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.8.204.189:39989)
0927 22:58:49.591799 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.591805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.597252 (+ 5447us) server_negotiation.cc:407] Connection header received
0927 22:58:49.597531 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.597538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.597641 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.597765 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.598725 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34882: BlockingRecv error: recv got EOF from 127.0.0.1:34882 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:58:49.649613 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.637471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.8.204.189:39989)
0927 22:58:49.637656 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.637664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.647953 (+ 10289us) server_negotiation.cc:407] Connection header received
0927 22:58:49.648164 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.648170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.648289 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.648434 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.649332 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34892: BlockingRecv error: recv got EOF from 127.0.0.1:34892 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 22:58:49.659464 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.657202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.8.204.189:39989)
0927 22:58:49.657403 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.657410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.657938 (+ 528us) server_negotiation.cc:407] Connection header received
0927 22:58:49.658130 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.658135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.658249 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.658377 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.659175 (+ 798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34902: BlockingRecv error: recv got EOF from 127.0.0.1:34902 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 22:58:49.673130 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.670936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.8.204.189:39989)
0927 22:58:49.671115 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.671122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.671659 (+ 537us) server_negotiation.cc:407] Connection header received
0927 22:58:49.671820 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.671828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.671929 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.672044 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.672889 (+ 845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34904: BlockingRecv error: recv got EOF from 127.0.0.1:34904 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 22:58:49.694772 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.692572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.8.204.189:39989)
0927 22:58:49.692741 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.692748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.693238 (+ 490us) server_negotiation.cc:407] Connection header received
0927 22:58:49.693423 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.693429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.693549 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.693671 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.694511 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34916: BlockingRecv error: recv got EOF from 127.0.0.1:34916 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 22:58:49.732818 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.730429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.8.204.189:39989)
0927 22:58:49.730626 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.730633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.731211 (+ 578us) server_negotiation.cc:407] Connection header received
0927 22:58:49.731456 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.731462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.731567 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.731694 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.732534 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34924: BlockingRecv error: recv got EOF from 127.0.0.1:34924 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 22:58:49.803011 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.800872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34940 (local address 127.8.204.189:39989)
0927 22:58:49.801097 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.801104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.801483 (+ 379us) server_negotiation.cc:407] Connection header received
0927 22:58:49.801686 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.801690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.801762 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.801850 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.802788 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34940: BlockingRecv error: recv got EOF from 127.0.0.1:34940 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:58:49.912684 32016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c86f4cd8e5f497f88c96496978c0186" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c86f4cd8e5f497f88c96496978c0186" member_type: VOTER last_known_addr { host: "127.8.204.190" port: 39771 } } peers { permanent_uuid: "222fd8279e1340de8073de6d4fb10870" member_type: VOTER last_known_addr { host: "127.8.204.189" port: 39989 } } peers { permanent_uuid: "61e059f90cd645e68807fb11ca1da57c" member_type: VOTER last_known_addr { host: "127.8.204.188" port: 42393 } } }
I20260927 22:58:49.913084 32016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c86f4cd8e5f497f88c96496978c0186 [sys.catalog]: This master's current role is: LEADER
W20260927 22:58:49.936303 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:49.934355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.8.204.189:39989)
0927 22:58:49.934586 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:49.934610 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:49.934861 (+ 251us) server_negotiation.cc:407] Connection header received
0927 22:58:49.935083 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:49.935089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:49.935185 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:49.935319 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:49.936065 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34942: BlockingRecv error: recv got EOF from 127.0.0.1:34942 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 22:58:50.199842 31915 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:58:50.197990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.8.204.189:39989)
0927 22:58:50.198234 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:50.198239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:50.198426 (+ 187us) server_negotiation.cc:407] Connection header received
0927 22:58:50.198629 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:50.198634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:50.198708 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:50.198823 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:58:50.199624 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34946: BlockingRecv error: recv got EOF from 127.0.0.1:34946 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:58:50.200650 32002 catalog_manager.cc:1259] Loaded cluster ID: 5a703d95238f47d3baa21c17310080be
I20260927 22:58:50.200747 32002 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: loading cluster ID for follower catalog manager: success
I20260927 22:58:50.203179 32002 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: acquiring CA information for follower catalog manager: success
I20260927 22:58:50.205027 32002 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 61e059f90cd645e68807fb11ca1da57c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:58:50.532078 32052 catalog_manager.cc:1259] Loaded cluster ID: 5a703d95238f47d3baa21c17310080be
I20260927 22:58:50.532225 32052 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: loading cluster ID for follower catalog manager: success
I20260927 22:58:50.534795 32052 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: acquiring CA information for follower catalog manager: success
I20260927 22:58:50.536669 32052 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 222fd8279e1340de8073de6d4fb10870: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:58:50.736176 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:50.729048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.8.204.189:39989)
0927 22:58:50.729908 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:50.729913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:50.729931 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:58:50.729995 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:50.729998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:50.730135 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:50.730237 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:50.730805 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.731741 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.733712 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.734168 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.734266 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:50.734849 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:50.735442 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:50.735562 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:50.735706 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:58:50.735789 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":548,"threads_started":1}
I20260927 22:58:50.753396 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:50.742816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.8.204.188:42393)
0927 22:58:50.747630 (+ 4814us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:50.747636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:50.747653 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:58:50.747741 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:50.747744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:50.747909 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:50.748083 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:50.748542 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.749278 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.750648 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.751307 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.751445 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:50.752026 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:50.752716 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:50.752870 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:50.752959 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 22:58:50.753043 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4706,"thread_start_us":4600,"threads_started":1}
I20260927 22:58:50.772625 32118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:50.759373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.8.204.189:39989)
0927 22:58:50.759538 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:50.759543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:50.764055 (+ 4512us) server_negotiation.cc:407] Connection header received
0927 22:58:50.764371 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:50.764379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:50.764498 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:50.764573 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:50.765445 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.766946 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.769804 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.770749 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.770883 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:50.771796 (+ 913us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:50.771936 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:50.771999 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:50.772223 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 22:58:50.772297 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:58:50.778410 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:50.759519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59002 (local address 127.8.204.190:39771)
0927 22:58:50.771992 (+ 12473us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:50.771998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:50.772019 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:58:50.772133 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:50.772141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:50.772377 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:50.772537 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:50.773370 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.774359 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.776052 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.776448 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.776605 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:50.777138 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:50.777582 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:50.777710 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:50.777894 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 22:58:50.777972 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12345,"thread_start_us":13128,"threads_started":1}
I20260927 22:58:50.778879 32124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:58:50.764079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.8.204.188:42393)
0927 22:58:50.764853 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0927 22:58:50.764861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:58:50.772106 (+ 7245us) server_negotiation.cc:407] Connection header received
0927 22:58:50.772531 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:58:50.772537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:58:50.772639 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:58:50.772705 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:58:50.773567 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.774613 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.776781 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:58:50.777249 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:58:50.777385 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:58:50.777865 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:58:50.778010 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:58:50.778102 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:58:50.778310 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:58:50.778385 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651}
I20260927 22:58:50.783654 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31770
I20260927 22:58:50.784814 31832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:50.899518 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31770
I20260927 22:58:50.919767 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31844
I20260927 22:58:50.920897 31906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:51.028379 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31844
I20260927 22:58:51.048616 9010 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu pid 31918
I20260927 22:58:51.049882 31980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:58:51.155782 9010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskoq4u9H/build/asan/bin/kudu with pid 31918
2026-09-27T22:58:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4814 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9140 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358199 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST